dis-aggregation as a vehicle for hyper-scalability in
TRANSCRIPT
![Page 1: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/1.jpg)
Dan Kilper
May 14, 2018
Dis-AggregationasaVehicleforHyper-ScalabilityinOpticalNetworks
![Page 2: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/2.jpg)
HyperScaleComputing
• Methodtoscaledatacentersto‘warehouse’sizes• 100k’sservers• Entiredatacenterbecomesthesystem
• Hardware/softwareseparationenabledDC-widecontrol• Tradeoffserverperformanceforcost&DCperformance
• Merchantsiliconopeneddoorfordatacenteroperatorstodesigntheirownservers
• EnabledholisticDCarchitectures• Computer‘integrators’bouncedbackbydesigningwholerackandpodsolutions
2
![Page 3: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/3.jpg)
DensificationofWirelessAccess
3
MetroCore
LongHaul
MetroCore
DistributionRings
PON
P2P
MicrowaveBH
AccessLink
WDMmmWave
CoreOCS
OLT
LongHaul
WDM-PON
AccessOCS
MacroRH
Micro/pico RH
BBU/DC
CORD/BBUpool
TodayFuture
• Networkoperatorsrequesting10k’sofaccesspointsineachUScity• Eachaccesspoint>10Gb/sbackhaul/fronthaul• Operatorsofferingwholewavelengthaccess(e.g.Pilot)
![Page 4: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/4.jpg)
WhatisDis-Aggregation?• Dis-aggregationiseconomicconcept
• Differentvendorsprovidepartsthatmakeupasystem
• Whethertodisaggregateisusuallydrivenbymarketandsupplychainconsiderations
• Dis-aggregationisanarchitectureconcept• Physicalorcontrolintegrationisseparated• Oftendeterminedbyperformancerequirements
4
![Page 5: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/5.jpg)
MarketDrivenComputerDis-AggregationEnabledHyperscale DCArchitecture
![Page 6: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/6.jpg)
TwoMainDriversforDis-Aggregation• Market
• Whenperformanceislessimportant• Whenscalabilityisneeded• Usemarketcompetitiontodrivedowncost
• Performance• Whencomponentperformanceismoreimportantthansystemperformance
• Whentechnologiesreachnewperformancelevelsenablingdisaggregation
• Usearchitectureenhancementstodrivedowncost
6
![Page 7: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/7.jpg)
ConventionalDataCenter
ToRServerServerServerServerServerServerServerServerServerServer
ToRServerServerServerServerServerServerServerServerServerServer
ToRServerServerServerServerServerServerServerServerServerServer
PackServersintoRacks
![Page 8: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/8.jpg)
ToR
Dis-aggregatedDataCenter
CPU/MEMCPU/MEMCPU/MEMCPU/MEMCPU/MEMCPU/MEMCPU/MEMCPU/MEMCPU/MEMCPU/MEM
ToRDISKDISKDISKDISKDISKDISKDISKDISKDISKDISK
ToRSSDSSDSSDSSDSSDSSDSSDSSDSSDSSD
ResourceperShelf
![Page 9: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/9.jpg)
Dis-aggregatedDataCenter
ToRCPU
MEMORYCPU
MEMORYCPU
MEMORYCPU
MEMORYCPU
MEMORY
ToRDISKDISKDISKDISKDISKDISKDISKDISKDISKDISK
ToRSSDSSDSSDSSDSSDSSDSSDSSDSSDSSD
![Page 10: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/10.jpg)
ToR
Dis-aggregatedDataCenter
CPUCPUCPUCPUCPUGPUGPUASICASICASIC
ToRDISKDISKDISKDISKDISKDISKDISKDISKDISKDISK
ToRMEMORYMEMORYMEMORYMEMORY
SSDSSDSSDSSDSSDSSD
![Page 11: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/11.jpg)
WhyDis-AggregateAgain?
• Ifyouhaveopticstothecomponentsthenincreaseinterconnectdistancesto~100m
• Latencyrequirementbecomesthelimitation
• Isserveroptimumcombinationofcpu/memory/disk/storage/NIC?
• Canvirtualizationbemoreefficientifremoveartificialboundariescreatedbyserverarchitecture?
• ServermemorylockedtoCPUs• Doesserverallowforbestnetworkarchitecture?• Optimizethermalmanagementtodevicerequirements
• Atshelfandracklevel
![Page 12: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/12.jpg)
ArchitectureDis-AggregationBenefits
12
![Page 13: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/13.jpg)
BringingOpticsInsidetheComputer• CPUIOBottleneck:
• NeedopticsforCPUtomemoryinterconnects• Itsgoingtobetherenomatterwhat
• Whataretheprospectsforscalingthisto10-100m?• D.A.B.MillerProc.IEEE2009
• Embeddedoptics:movingtheNIContotheboard• ExpandingtheNICandintegratingitonboard
• DataCenterOpticalNetworks• Ifyouhaveanetwork,whynotdis-aggregate?
![Page 14: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/14.jpg)
EmbeddedOptics
CPU
NIC
NIC
HighCapacityElectricalInterconnects
Startstolooklikeanopticalline
card…
![Page 15: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/15.jpg)
Dis-AggregatingOpticalSystems
15
![Page 16: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/16.jpg)
SomeHistory• Late90’s:MCI/Globecom triedtobuildtheirownsystemsfromcomponents
• ~2000:UnifiedcontrolplaneattempttomergecontrolofopticalsystemsintoL3control
• GMPLS/MPLSwasresult• Mid00’s:JDSU/Nortelintroduce‘generic’ROADMbuildingblocksystems
• Late00’s:Coherenttransceiverschangesystemengineering(nodispersionmaps,PMD)
• Early10’s:Enterprises/DCoperatorsbuildtheirownopticalnetworks
• 2020:5Giscoming!
16
![Page 17: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/17.jpg)
OpticalSystemVendors• HistoricallyopticalsystemvendorsNOT‘systemintegrators’
• Opticalsystemsareengineeredproducts• Componentsandsub-systemshighlyspecifictosystemdesign• Tightlycoupledhardwareandsoftwaredesign• LongR&Dandtestcyclestodevelopproduct
• Keyquestion:Canopticalsystemvendorsmovetosystemintegratormodel?
• SimilartoDellorHP• Oroperatingsystemmodel?e.g.Microsoft
17
![Page 18: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/18.jpg)
Hyperscale Attributes
• Largenumbersofaccesspoints(ROADMnodes)• Gofrom100’spercityto10k-100kpercity• Designedatthenetworkleveltoachievescalability
• Unifiedandscalablesoftwarecontrol• Remove‘siloing’– hardwaretiedtosoftware(operatingsystem)
18
![Page 19: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/19.jpg)
ProprietaryOpticalSystems
19
ROADM
ROADM
ROADM
ROADM
ROADM
ROADM
ROADM
NetworkOrchestrator/OperatingSystem
OLSControl OLSControl
OLSManagementSystem OLSManagementSystem
OTN/L2
OTN/L2
OTN/L2OTN/L2
OTN/L2L2/L3
![Page 20: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/20.jpg)
TransceiverDisaggregation(Alienls)
20
ROADM
ROADM
ROADM
ROADM
ROADM
ROADM
ROADM
NetworkOrchestrator/OperatingSystem
OLSControl OLSControl
OLSManagementSystem OLSManagementSystem
OTN/L2
OTN/L2
OTN/L2OTN/L2
OTN/L2L2/L3
![Page 21: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/21.jpg)
Whitebox/openROADM Systems
21
ROADM
ROADM
ROADM
ROADM
ROADM
ROADM
NetworkOrchestrator/OperatingSystem
Menara:BuiltinOTN
![Page 22: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/22.jpg)
ComputerSystemIntegration• Stillvalueinmatchingcomponentstomotherboardandgoodsystemdesignprinciples
22
![Page 23: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/23.jpg)
Whitebox/OpenOpticalNetworks
23
ROADM
ROADM
ROADM
ROADM
ROADM
ROADM
ROADM
NetworkOrchestrator/OperatingSystem
OLSControl&ManagementSystem
![Page 24: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/24.jpg)
CostModels:Where’stheSavings?
24Riccardi,et.al.JLT2018
![Page 25: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/25.jpg)
TransceiverSavings:AvoidRegens
• Withalmostnoregeneration
25J.Santoset.al.JOCN2018
![Page 26: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/26.jpg)
WithRegeneration
• Disaggregationpenalty&networkdomainsmakeadifference
26J.Santoset.al.JOCN2018
![Page 27: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/27.jpg)
TransmissionReach
27
InMetro&datacenternetworks:Distance=#Hops2000km~20hops
Bosco,et.al.JLT2011
Higherordermodulation
![Page 28: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/28.jpg)
OpticalPowerDynamics• OpticalpowerdynamicsinOADMringnetwork
• Simulations&modelingofchannelpoweroscillationsandinstability
• L.PavelAutomatica 2004• Gorinevsky &FarberJLT2004
28
SustainedOscillationsoverLongPeriods
![Page 29: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/29.jpg)
DynamicDomainPowerControlAlgorithm• Powerdriftsovertimeandnewchannelsareprovisioned:needperiodicpowercontroltostaywithinmargins
• Adjustnodesinparallelwithin‘optically’isolateddomains
• Nodeorderingbasedonchannelroutes
29
[1]
A B
[1,2,3,4] [1,4]
[3]
[2]C
D
[3]
[4]
[ ]
Wait for Chn 1
Ready to Adjust
[i,j,k] = channels adjusting upstream
4
511
12
22
23
3130
29
28
27
2625
24
21
13 15
20
18
19
1614
109
17
87
123
6
Kilper&WhiteOFC2007
![Page 30: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/30.jpg)
Objectives:• tunabledrop(reject)
4-channeltunableadd4+1channelVOA
• 100,000timessmaller• approx.250mW• nomovingparts
WDM Network Node on-a-Chip:Lower performance, but much lower cost
R.Aguinaldo,H.Grant,S.Mookherjea(UCSD)+Sandia
Channel 36 Channel 35
Channel 34 Channel 33
100 ps
A B
C D
1.3mmx0.52mm
8fiberV-groovearray
4x10Gbps addedfromindividualINfiberstocommonOUT
Common (23 ch)OUT (23 ch)Diag (“Test”)A B, C, D (in)
In/Out
all channels on ITU-T 100 GHz grid 30
![Page 31: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/31.jpg)
SystemLevelIssues• Transceiver&systemperformanceinteractions
• Biggerproblemforbleedingedgeperformance• Transceiverscomplexsystemsontheirown
• Blockingbadcornercases• Handlingthewiderangeofsystemfunctions• Systemtestingpullsinmargins
• Toomanyuncertainties
• Controldynamics• Opticalpowerdynamics
31
![Page 32: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/32.jpg)
ResearchQuestions• Atwhatmetroreach(numberofnodehops)dothedifferentdisaggregationmodelsbecomeproblematic?Forwhichtransceivertypes?
• Howdoesphysicallayersoftwarecontrolscalewithnumberofnodes?
• DICONETandotherexamplesforlonghaulneedtobeadaptedhere
• Needtoolstodevelopandtestcontrolatscale(seenexttalk)
• Whatcomponentscanbescaledtoverylargenumbers?
• Needintegratedphotonics
32
![Page 33: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/33.jpg)
Conclusions• Computingsystemsaregoingthroughmultipleroundsofdisaggregationinordertocontinuehyperscale growth
• Marketand/orperformancedrivenarchitecturalchange
• 5Gcreatespotentialforopticalsystemstojumptohyperscale models
• Notjustaboutopeningcompetitionfortransceivers,needfullnetworkdesignforhyperscale growth
• Transmissionengineeringremainsanobstacle• Hardware&Software• Neednewtoolstackleproblem(machinelearning?)
• Savingsneedtocomefromhighvolumes:needtothinkhyperscale
33
![Page 34: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/34.jpg)
www.cian-erc.org
CenterforDis-IntegratedandDis-AggregatedNetworks
![Page 35: Dis-Aggregation as a Vehicle for Hyper-Scalability in](https://reader031.vdocuments.net/reader031/viewer/2022011808/61d442d822aa42510d3b6305/html5/thumbnails/35.jpg)
Thank You
Our Group:https://wp.optics.arizona.edu/dkilper/
CIAN:www.cian-erc.org
35