alternative switching technologies: wireless datacenters
TRANSCRIPT
![Page 1: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/1.jpg)
Alternative Switching Technologies: Wireless Datacenters
Hakim WeatherspoonAssistant Professor, Dept of Computer Science
CS 5413: High Performance Systems and NetworkingOctober 22, 2014
Slides from the “On the Feasibility of Completely Wireless Datacenters” at the ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS),October 2012.
![Page 2: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/2.jpg)
Goals for Today• On the Feasibility of Completely Wireless
Datacenters– J. Y. Shin, E. G. Sirer, H. Weatherspoon, and D. Kirovski,
IEEE/ACM Transactions on Networking (ToN), Volume 21, Issue 5 (October 2013), pages 1666-1680.
![Page 3: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/3.jpg)
Conventional Datacenter
…Top of Rack Switch
Aggregate Switch
Core Switch
…
![Page 4: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/4.jpg)
Going Completely Wireless• Opportunities
– Low maintenance : no wires– Low power: no large switches– Low cost: all of the above
– Fault tolerant: multiple network paths– High performance: multiple network paths
Which wireless technology?
![Page 5: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/5.jpg)
60GHz Wireless Technology
• Short range – Attenuated by oxygen
molecules
• Directional– Narrow beam
• High bandwidth – Several to over 10Gbps
• License free– Has been available for
many yearsWhy now?• CMOS Integration
- Size < dime- Manufacturing cost < $1
[Pinel ‘09]
7 m
m
5 mm
Rx Tx
6
![Page 6: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/6.jpg)
60 GHz Antenna Model
• One directional– Signal angle between
25° and 45°– Maximum range < 10 m– No beam steering
• Bandwidth < 15Gbps – TDMA (TDD) – FDMA (FDD)
• Power at 0.1 – 0.3W
How to integrate to datacenters?
![Page 7: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/7.jpg)
Designing Wireless Datacenters• Challenges
– How should transceivers and racks be oriented?– How should the network be architected?– Interference of densely populated transceivers?
![Page 8: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/8.jpg)
Completely Wireless Datacenters
• Motivation• Cayley Wireless Datacenters
– Transceiver placement and topology• Server and rack designs
– Network architecture• MAC protocols and routing
• Evaluation– Physical Validation: Interference measurements– Performance and power
• Future• Conclusion
![Page 9: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/9.jpg)
Transceiver Placement: Server and Rack Design
• Rack • Server
Intra-rack space
Inter-rack space
2D View
3D View
3-way switch (ASIC design)
How do racks communicate with each other?
![Page 10: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/10.jpg)
Cayley Network Architecture: Topology
![Page 11: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/11.jpg)
Masked Node Problem and MAC• Most nodes are hidden terminals to others
– Multiple (>5) directional antennae => Masked node problem
– Collisions can occur
• Dual busy tone multiple access [Hass’02]– Out of band tone to preserve channels– Use of FDD/TDD channels as the tone
12
![Page 12: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/12.jpg)
Cayley Network Architecture: Routing
• Geographical Routing• Inter rack
– Diagonal XYZ routing
• Turn within rack– Shortest path turning
• Within dst rack to dstserver– Up down to dst story– Shortest path to dst server
1818181818181818181818181818181818181818S 1818181818181818181818181818181818181818 1818181818181818181818181818181818181818
1818181818181818181818181818181818181818D18181818181818181818181818181818181818181818181818181818181818181818181818181818
181818181818181818181818181818181818181818181818181818181818181818181818181818181818181818181818181818181818181818181818
1818181818181818181818181818181818181818
![Page 13: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/13.jpg)
Completely Wireless Datacenters
• Motivation• Cayley Wireless Datacenters
– Transceiver placement and topology• Server and rack designs
– Network architecture• MAC protocols and routing
• Evaluation– Physical validation: Interference measurements– Performance and power
• Future• Conclusion
![Page 14: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/14.jpg)
Hardware Setup for Physical Validation
• Use of a conservative platform• Real-size datacenter floor plan setup• Validation of all possible interferences
Intra-rack communications Inter-rack communications
![Page 15: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/15.jpg)
Physical Validation: Interference Evaluation(Signal angle θ = 15° )
-80
-75
-70
-65
-60
-55
-50
-45
-40
10 9 8 7 6 5 4 3 2 1
RSS
(dB)
Server ID of Rx
Intra-Rack Space (Tx on server 0)
Error free Default noise
![Page 16: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/16.jpg)
Physical Validation: Interference Evaluation(Signal angle θ = 15° )
-80-75-70-65-60-55-50-45-40
15 14 13 12 11 10
RSS
(dB)
Server ID of Rx on Rack C
Non-Adjacent Inter-Rack Space (Tx on Rack D)
Error free Default noise
Tx: server 2
-80-75-70-65-60-55-50-45-40
10 9 8 7 6
RSS
(dB)
Server ID of Rx on Rack A
Orthogonal Inter-Rack Space (Tx on Rack D)
Error free Default noiseTx: server 0
-80-75-70-65-60-55-50-45-40
15 14 13 12 11 10
RSS
(dB)
Server ID of Rx on Rack B
Diagonal Inter-Rack Space (Tx on Server 2 of Rack D)
Error free Default noise
Edge of signal:can be eliminated
-80-75-70-65-60-55-50-45-40
10 9 8 7 6
RSS
(dB)
Server ID of Rx on Rack A
Orthogonal Inter-Rack Space (Tx on Rack D)
Error free Default noiseTx: server 0 Tx: server 1
-80-75-70-65-60-55-50-45-40
10 9 8 7 6
RSS
(dB)
Server ID of Rx on Rack A
Orthogonal Inter-Rack Space (Tx on Rack D)
Error free Default noiseTx: server 0 Tx: server 1Tx: server 2
-80-75-70-65-60-55-50-45-40
15 14 13 12 11 10
RSS
(dB)
Server ID of Rx on Rack C
Non-Adjacent Inter-Rack Space (Tx on Rack D)
Error free Default noise
Tx: server 2 Tx: server 3
-80-75-70-65-60-55-50-45-40
15 14 13 12 11 10
RSS
(dB)
Server ID of Rx on Rack C
Non-Adjacent Inter-Rack Space (Tx on Rack D)
Error free Default noiseTx: server 2 Tx: server 3Tx: server 4
Potential Interference:
can be blocked using conductor
curtains
![Page 17: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/17.jpg)
Evaluation• Performance: How well does a Cayley datacenter
perform and scale?– Bandwidth and latency
• Failure tolerance: How well can a Cayleydatacenter handle failures?– Server, story, and rack failure
• Power: How much power does a Cayleydatacenter consume compared to wired datacenters
![Page 18: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/18.jpg)
• Simulate 10K server datacenter – Packet level: routing, MAC protocol, switching delay, bandwidth
• Conventional datacenter (CDC)– 3 Layers of oversubscribed switches (ToR, AS, CS)
• (1, 5, 1), (1, 7, 1) and (2, 5, 1)• Latency: 3-6us switching delay• Bandwidth: 1Gbps server
• FAT-tree: Equivalent to CDC (1,1,1)• Cayley wireless datacenter
– 10Gbps bandwidth– 1 Transceiver covers 7 to 8 others– Signal spreading angle of 25°– Low latency Y-switch (<< 1us)
Evaluation Setup
Top of Rack
Aggregate
Core
10
10
2
(1,5,1)
10
10
1.4
(1,7,1)
10
5
1
(2,5,1)
10
10
10
(1,1,1)
![Page 19: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/19.jpg)
Evaluation Setup
• Uniform random– Src and dst randomly selected in entire datacenter
• MapReduce– Src sends msg to servers in same row of rack– Receiver sends msg to servers in same column of rack– Receivers send msg to servers inside same pod with
50% probability
![Page 20: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/20.jpg)
Cayley datacenters have the most bandwidth
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
Uniform RandHops: CDC < 6, Cayley > 11
MapReduceHops: CDC < 6, Cayley > 8
Maximum Aggregate Bandwidth Normalized to Fat-tree
fat-tree CDC 151CDC 171 CDC 251Cayley
Bandwidth
s
• Burst of 500 x 1KB packets per server sent
![Page 21: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/21.jpg)
Latency• Uniform random benchmark
• MapReduce benchmark
Cayley datacenters typically performs the best
0
50
100
150
200
100 200 300 400 500
Late
ncy
(us)
Packet Injection Rate (Packet/Second/Server)
Uniform Random (4KB Packet)
fat-tree CDC 251
CDC 171 CDC 151
Cayley
0
2000
4000
6000
8000
10000
100 200 300 400 500
Late
ncy
(us)
Packet Injection Rate (Packet/Second/Server)
Uniform Random (16KB Packet)
0
200
400
600
100 200 300 400 500
Late
ncy
(us)
Packet Injection Rate (Packet/Second/Server)
MapReduce (4KB Packet)
0500
1000150020002500
100 200 300 400 500
Late
ncy
(us)
Packet Injection Rate (Packet/Second/Server)
MapReduce (16KB Packet)
![Page 22: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/22.jpg)
Fault Tolerance
Cayley datacenters are extremely fault tolerant
0
20
40
60
80
100
0 10 20 30 40 50 60 70 80 90
Pres
erve
d co
nnec
tivity
(%)
Failed components (%)
Preserved connectivity among live nodes
Node
Story
Rack
25% 55% 77%99%
![Page 23: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/23.jpg)
Power Consumption to Connect 10K Servers
• Conventional datacenter (CDC) *
– Depending on the oversubscription rate 58KW to 72KW
• Cayley datacenter– Transceivers consume < 0.3W
– Maximum power consumption: 6KW
• Less than 1/10 of CDC power consumption
Switch Type Typical Power
Top of rack switch (ToR) 176W
Aggregation switch (AS) 350W
Core switch (CS) 611W
* Cost and spec of Cisco 4000, 5000, 7000 series switches
![Page 24: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/24.jpg)
Discussion and Future Work
• Only scratched the surface– How far can wireless datacenters go with no wires?
• Need larger experiment/testbed– Interference and performance of densely connected
datacenter?
• Scaling to large datacenters (>100K servers)?• Scaling to higher bandwidth (> 10Gbps)?
![Page 25: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/25.jpg)
Conclusion
• Completely wireless datacenters can be feasible• Cayley wireless datacenters exhibit
– Low maintenance– High performance– Fault tolerant– Low power– Low cost
![Page 26: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/26.jpg)
References• S. Pinel, P. Sen, S. Sarkar, B. Perumana, D. Dawn, D.
Yeh, F. Barale, M. Leung, E. Juntunen, P. Vadivelu, K. Chuang, P. Melet, G. Iyer, and J. Laskar. 60GHz single-chip CMOS digital radios and phased array solutions for gaming and connectivity. IEEE Journal on Selected Areas in Communications, 27(8), 2009.
• Z.J. Hass and J. Deng. Dual busy tone multiple access (DBTMA)-a multiple access control scheme for ad hoc networks. IEEE Transactions on Communications, 50(6), 2002.
• PEPPM. Cisco Current Price List. http://www.peppm.org/Products/cisco/price.pdf, 2012.
27
![Page 27: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/27.jpg)
Link Technology Modifications Required
WorkingPrototype
Helios(SIGCOMM ‘10)
Optics w/ WDM10G-180G (CWDM)10G-400G (DWDM)
Switch Software Glimmerglass, Fulcrum
c-Through(SIGCOMM ’10)
Optics (10G) Host OS Emulation
Flyways(SIGCOMM ‘11, HotNets ‘09)
Wireless (1G, 10m) Unspecified
IBM System-S(GLOBECOM ‘09)
Optics (10G) Host Application;Specific to StreamProcessing
Calient,Nortel
HPC(SC ‘05)
Optics (10G) Host NIC Hardware
Related Work
![Page 28: Alternative Switching Technologies: Wireless Datacenters](https://reader036.vdocuments.net/reader036/viewer/2022081623/6157d964ce5a9d02d46fca88/html5/thumbnails/28.jpg)
Before Next time• Project Interim report
– Due Monday, October 27.– And meet with groups, TA, and professor
• Lab3 – Packet filter/sniffer– Due yesterday, Tuesday, October 21. But, 24 hour grace period.
• Lab1/2 redux due Friday, October 24• Fractus Upgrade: SAVE ALL YOUR DATA
– Fractus will be upgraded from October 28th to 30th
– Can use Red Cloud during upgrade period, then switch back to Fractus
• Required review and reading for Wednesday, October 22– Data center TCP (DCTCP), M. Alizadeh, A. Greenberg, D. A. Maltz, J. Padhye, P. Patel, B.
Prabhakar, S. Sengupta, and M. Sridharan. ACM SIGCOMM Computer Communication Review (CCR), Volume 40, Issue 4 (October 2010), pages 63-74.
– http://dl.acm.org/citation.cfm?id=1851192– http://www.sigcomm.org/sites/default/files/ccr/papers/2010/October/1851275-1851192.pdf
• Check piazza: http://piazza.com/cornell/fall2014/cs5413• Check website for updated schedule