PICMG?3.2R1.0

InfiniBand?forAdvancedTCA?SystemsShortFormSpecification

January22,2003

FORINFORMATIONONLY;DONOTATTEMPTTODESIGNFROMTHISDOCUMENT

NOTE:ThisshortformspecificationisasubsetoftheInfiniBand?forAdvancedTCA?Systemsspecification,PICMG3.2R1.0.ForcompleteguidelinesonthedesignofInfiniBand?forAdvancedTCA?implementations,thefullspecificationisrequired.

ToorderafullcopyofthePICMG3.2specification,goto

,orcontactthePCIIndustrialComputerManufacturersGroupat401EdgewaterPlace,Suite500,Wakefield,Mass.,01880.Phone781-246-9318,fax781-224-1239,email

info@

.

Copyright2003PCIIndustrialComputerManufacturersGroup.

PICMGdisclaimsallwarrantiesandliabilityfortheuseofthisdocumentandtheinformationcontainedherein,andassumesnoresponsibilityforanyerrorsoromissionsthatmayappearinthisdocument,norisPICMGresponsibleforanyincidentalorconsequentialdamagesresultingfromtheuseofanydatacontainedinthisdocument,nordoesPICMGassumeanyresponsibilitytoupdateorcorrectanyinformationinthispublication.

PICMG,CompactPCI,andthePICMGandCompactPCIlogosareregisteredtrademarks,andAdvancedTCAandATCAandtheAdvancedTCAandATCAlogosaretrademarksofthePCIIndustrialComputerManufacturersGroup.InfiniBandisaregisteredtrademarkoftheInfiniBandTradeAssociation.Allotherbrandorproductnamesmaybetrademarksorregisteredtrademarksoftheirrespectiveholders.

FORINFORMATIONONLY;

DONOTATTEMPTTODESIGNFROMTHISDOCUMENT

Ethernet/FibreChannelforAdvancedTCA?SystemsShortFormSpecification

Introduction

ThecompletePICMG3.2SpecificationisamemberofthePICMG3.0AdvancedTCA?seriesofspecifications.Thebasespecificationinthefamily,PICMG3.0,definesaBoardandShelfarchitectureformodularcomputingandcommunicationscomponents.TheBoardsshareacommonBackplaneforpoint-to-pointdifferentialconnectivitythatsupportsmanycommonhigh-speedswitchedfabrictechnologies.PICMG3.2specifiestheusageofInfiniBand?overthisbackplanefabric.

Scope

ThescopeofthiseffortincludesdesignrulesandguidelinesforimplementationofInfiniBandBoardsbasedonthePICMG3.0specification.

Objectives

Thespecificationinthefull3.2documentiswhollyderivedfrom,andisdependentupon,thePICMG3.0BaseSpecification.ItisnotintendedtostandaloneortobeusedseparatelyfromtheBaseSpecification.

ThecompletePICMG3.2documentbuildsuponthePICMG3.0BaseSpecificationandtheInfiniBandSpecificationstomeetthefollowingobjectives:

DefinehowInfiniBandsignalingistobeusedovertheFabricInterfacedefinedinthePICMG3.0Specification

Provideguidelinesfortheuseof1X,4X,and12XInfiniBandlinks

ProvideguidelinesfortheuseofInfiniBandin-bandmanagement

ProviderecommendationsandguidelinesforutilizationofadvancedInfiniBandfeaturesthatincreaseperformanceandinteroperability

TheguidelinesanddesignrulessetforthinthecompletespecificationshallbeconsistentwiththeapplicablesectionsoftheInfiniBandSpecificationversionasreferencedinSection1.3ofthecomplete3.2document.PICMG

3.2productswillcomplywithInfiniBandSignal,Link,Transport,andManagementlayersinordertomaximizeinteroperabilitywithotherInfiniBandhardwareandsoftwareproducts.

FORINFORMATIONONLY;

DONOTATTEMPTTODESIGNFROMTHISDOCUMENT

Ethernet/FibreChannelforAdvancedTCA?SystemsShortFormSpecification

Overview

ThecompletePICMG3.2documentspecifiesanopenarchitectureformodularcomputingandcommunicationscomponentsbasedonInfiniBandtechnology.ThefullPICMG3.2isasubsidiaryofthePICMG3.0BaseSpecification,andtogethertheyspecifyanInfiniBand-basedarchitectureforconvergingcommunicationsanddatanetworkingapplications.ThereareseveraladvantagestousingInfiniBandasabackplanefabric:

Improvedbackplanecapacity:InfiniBandsupports10Gb/sbi-directionalconnectionstodaywithstandards-basedsilicon.Thisreachesacapacityof2.4Tb/sinasinglePICMG3.2Shelfusing16Boardsinafullmeshbackplanetopology.

ImprovedQualityofService:ThevirtuallaneandservicelevelmechanismsofInfiniBandallowthestrictsegregationofdifferentclassesoftraffic.Thispermitsthecreationofequipmentthatsupportsdifferentapplicationswithoutriskoflowerprioritytrafficdelayinghigherprioritytraffic.

HardwareTransport:InfiniBandprovidesreliableconnectionsinhardwarewhicheliminatesthesoftwareoverheadofatransportprotocolstack.Thisnotonlyenableslow-latencyandhighthroughputforserverclusteringandstorage,butenablessimplifiedmanagementandmulti-protocolsupportforrealtimetraffic.

Multi-protocolsupport:SincemanydifferentprotocolscanbeeasilymappedontoandtransportedbyInfiniBand,aPICMG3.2systemcansupportIP,ATM,TDM,dataacquisitionprotocols,andmixedprotocolsystems.

Integrationofserversandstorage:InfiniBandwasdevelopedasastandardRDMA-basedclusteringinterconnectforserversandStorageAreaNetworks(SANs).

Simplifiedmulti-Shelfsystems:ThesamesimplifiedcapabilitiesforinterconnectwithserversandnetworkedstoragealsofacilitatetheinterconnectionofmultiplePICMG3.2shelveswithInfiniBandcablestocreateverylarge,multi-Shelfsystemsthatbehaveassinglelogicalsystems.

Improvedmanagement:TheextensivenetworkmanagementcapabilitiesavailableinInfiniBandworkinconcertwiththePICMG3.0ShelfManagementinfrastructuretoprovideasimplifiedandhighlyavailablemanagementframework.

Theseadvantagesbringsignificantimprovementstothedevelopmentoftelecommunicationsanddataprocessingsystems.

ApplicationExample

Figure1showsanexampleapplicationofPICMG3.2toamulti-servicetelecom/datacomnetworkelementintendedfordeploymentinatelecommunicationscentraloffice,datacenter,orworkgroupmachineroom.Itsfunctionsincludeuplinkstoseveralbackbonenetworktypes(IP,ATMandSONET/TDM),severaldifferentuserinterfaces(copperEthernetandfiber),signalprocessing,serverprocessors,andstoragemanagement.

Figure1.ShelfLevelBlockDiagram

PICMG3.2Backplane

IP

/HUB

IP

/HUB

ATM

ATM

SONET

TDM

SONET

TDM

1

0

0

0

BT

1

0

0

0

BT

FIBER

FIBER

DSP

DSP

PROCESSOR

PROCESSOR

SAN

SAN

Multi-Protocol

BackboneInterfaces

LocalClientInterfaces

ProcessingResources

Storage

Interfaces

EighttypesofBoardsaredeployedinaredundantmannertoenablehighavailabilityandreliability.BoardsthathavespecialfeaturesandperformancerequirementswouldlikelybecustomdesignedbyaTelecommunicationsEquipmentManufacturer(TEM).However,manyoftheBoardswillbeavailableoff-the-shelffromcommercialvendorsinthecomputingandtelecommunicationsindustry.Server,storage,andEthernetBoards,forexample,willbewidelyavailableforenterpriseanddatacenterapplications.Thisoffersasignificanttime-to-marketandcostsavingsadvantagestotheTEM.

ThefollowingbrieflydescribesthedifferentBoardtypes:

TheIP/HUBBoardsprovideswitchingandroutingbetweenanexternalIPWANandtheInfiniBandfabric.AtypicalconfigurationmightincludeanInfiniBandswitchfor15backplaneportswithmultipleGigabitEthernetportsonthefaceplate.WirespeedpacketprocessingbetweenEthernetandInfiniBandwouldsupporttheIP-over-IBprotocolforroutingIPtootherBoardsinthesystem,andtheSocketsDirectProtocolforterminatingtheconnectiontoaninternalInfiniBandserverorDSPBoard.Firewallandsecurityprocessingcanalsobesupportedbyon-boardprocessorsorotherprocessorBoardsinthesystem.IfaDualStarTopologyisused,thisBoardwouldalsoprovidethePICMG3.0Hubfunctionality.

TheATMBoardsadapttheInfiniBandprotocolfoundontheBackplanetoATM,andprovideATMinterfaces(probablymappedoverSONET)toATMcorenetworks.

SONET/TDMBoardssupportTimeDivisionMultiplexed,channelizedtrafficsuchasvoicetrunks.TheyincludetheprotocolconversionlogicneededtomaptimeslotstotheInfiniBandBackplaneinterconnect.VoicetrafficmaybesenttoothervoiceBoardsforcallswitchingfunctions,ormaybesenttotheDSPorserverBoardsforVOIPpacketprocessingandmessagingapplications.

The1000BASE-TlocalclientinterfaceBoardssupportLAN/MANtraffictolocaldestinationssuchascustomerpremiselastmileordesktopcomputerswithinthebuilding.ThearchitecturemightbesimilartotheIP/HUBinterfaceBoard,exceptwithoutthefirewallandsecurityfunctions.

FiberinterfaceBoardsperformasimilarfunctionsasthe1000BASE-TinterfacesexceptfortheuseoffibertoprovidelongerreachforapplicationslikeFibertotheHomeorMetropolitanOpticalNetworks.

TheDSPBoardsprovidesignalprocessingcapacityforapplicationsrequiringvocoding,compression,encryption,conferencebridging,andvideoencoding.

FORINFORMATIONONLY;

DONOTATTEMPTTODESIGNFROMTHISDOCUMENT

Ethernet/FibreChannelforAdvancedTCA?SystemsShortFormSpecification

TheprocessorBoardsprovidestandardgeneral-purposeRISCorCISCcomputersthatareoftenusedasservers.Theycanprovidehighperformancedatabaseapplicationsaswellasruncallprocessing,protocolprocessing,ornetworkmanagementapplications.

Finally,theSANBoardsaretheinterfacestoStorageAreaNetworks.ThesemightbeexternaldiskarrayshelvesprovidingmanyterabytesofstorageusingInfiniBand’sabilitytobridgeeasilytoFibreChannel.IfInfiniBandisusedtointerconnectthediskarrayswiththePICMG3.2Shelf,theSANBoardshaveverysimpledesigns(sincethereisnoneedtoadaptthenativeInfiniBandprotocolfromtheBackplane).AnalternativeimplementationcouldmountseverallowprofilediskdrivesdirectlyontheBoards,providingcapacityofhundredsofgigabytesintegraltothePICMG3.2Shelf.BoardsofsimilararchitecturetotheSANBoardprovideShelf-to-ShelfIBlinkssupportingmulti-Shelfsystems.

TopologiesandBandwidth

ThePICMG3.2Backplaneisthecoreofthisreferencearchitectureandoffersachoiceoftopologies,linkspeeds,protocols,andservicessupportedonthesystem.

ThetwomostcommonfabrictopologychoicesareDualStarandMesh.AdualstarsystemusestwoHubBoardsinLogicalSlots1and2(labeledIP/HUBinFigure1)andNodeBoards(eachwithtwoInfiniBandlinks)installedintheremaining14slots.A16SlotDual-StarSystem(14NodeSlots)using4XInfiniBandlinksuses28linkstocreatethefabric,oratotalbandwidthof[28*10Gb/s=]280Gb/s.Using1Xlinks,thismeshprovidesarawcapacityof[28*2.5Gb/s=]70Gb/s.

AMeshtopologyutilizesallavailablelinksinallitsslots.MeshEnabledBoardshaveInfiniBandswitchingcapacityforallpossiblelinks.ForexampleaMeshEnabledBoardina16slotsystemwillhave15InfiniBandlinkstotheBackplane(onetoeachoftheotherBoards).A4XMeshreachesthetheoreticalmaximumcapacityof[16*15*10Gb/s=]2.4Tb/s.A1XMeshiscapableof[16boards*15links*2.5Gb/s=]600Gb/stheoreticalmaximumcapacity.

InfiniBanduses8B/10Bencoding,thereforeusablebandwidthis80%oftheabovequotedlinerates.Also,usablepayloadbandwidthwillhavetobederatedtoaccountfortheInfiniBandprotocoloverhead.

PICMG3.0BackplanesalsoincludetwoBaseInterfaceChannels,threeSynchronizationClockInterfaces,twoIPMBchannels,andtheUpdateChannelInterfacetoallslotsforusebyadvancedtelecommunicationsBoards.Thesefeatureswillnotbediscussedindetailinthisdocument,butPICMG3.2systemscanmakeuseofthemasneeded.

BoardDesignExample

Figure2isablockdiagramofaPICMG3.2Boardusinga1Xmeshtopologycapableofsupporting2.5Gb/sofrawthroughputtoeachof15otherBoards.ThisdesignisasupersetofthedesignsrequiredfortheHubBoardsorNodeBoardsneededwithaDualStarTopology.AfourmezzaninedesignisshowntoillustratetheflexibilityofsupportingmultiplekindsofI/Oandprocessingresourcesonauniversalboardtype.Thedesignwouldbesimilarifthemezzaninesweredeletedandtheprimefunctions(processors,DSPs,I/O,etc.)weremounteddirectlyontheBoard.

DualStarNodeBoardswouldbesimilarindesigntothisexample,butsimplified,astheInfiniBandswitchfunctionandmuchofthenetworkprocessingcapabilitycanbedeleted.TwoHostChannelAdapterfunctionsarerequiredonDualStarNodeBoards,oneforeachlinktotheduplicatedHubBoards.

DualStarHubBoardswouldalsobesimilartothisexample,buttheiremphasiswouldbeontheInfiniBandSwitchandNetworkProcessorfunctionality.TheControllerCoreAssetandI/OandProcessingblocksareoptionalinthecaseofHubBoards,andareneededonlyinhighdensitysystemsthatprovideBoardfunctionalitybeyondsimplehubfunctionsonLogicalSlots1and2.

TheMeshreferencedesigncanalsobeextendedtosupport4Xand12XInfiniBandlinksbyusingappropriateInfiniBandswitchcomponentswithhighercapacityBackplaneconnections,andincreasingtheHostChannelAdapterandnetworkprocessingthroughputaccordingly.Also,usinga12XlinksrequiresthattheBackplaneroutingsupportsreplicatedmeshes,perSection.

Figure2.ExamplePICMG3.2MeshBoardDesign

Backplanecoreasset

PICMG3.2BackplaneInterface

BufferandTableRAM

16X16

InfiniBandSwitch

MeshconnectionHostChannelAdapter/

NetworkProcessor

Power

Control

64bitPCIBus

BoardLevelProcessor

BufferRAM

Peripherals

RAM

ControllerCoreAsset

I/O&Processing

MEZ#1DSP

MEZ#2

Processor

MEZ#3

I/O

...

MEZ#4

Storage

ThePICMG3.2Backplanetrafficentersa16X16InfiniBandswitchcomponentthatperformsthedistributedswitchingfunctionsfortheShelf.ItsprimarypurposeistoselectanddirectthebackplanetrafficthatistobeprocessedbythisBoard.IfthisBoardactsastheswitchHubforadualstarbackplanetopology(installedinlogicalslot1or2),thisswitchcomponentrepresentsthemainHubfunctionforanentireShelf.BecauseofthefocusedloadonthesixteenthlinkbetweentheswitchandtheHostChannelAdapter,thislinkshouldsupporthighertrafficcapacity.ForaDualStartopologythislinkmaybescaleddowndependingontheoutgoingtrafficrequirements.

SincethisinterfaceoftenrepresentsthesystemlevelbandwidthbottleneckforaPICMG3.2Shelf,itisadvisabletomakeitashighacapacityaspractical.

NextinthesignalprocessingchainisanInfiniBandHostChannelAdapterfunction.ThisterminatestheInfiniBandprotocolfromthebackplanelinks,andmakesthepayloadavailabletotherestoftheBoard.

Anetworkprocessorfunctionisassociatedwit