![Page 1: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/1.jpg)
1
High PerformanceSwitching and RoutingTelecom Center Workshop: Sept 4, 1997.
OR Project Group II:Packet Buffer Proposal
Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz
E-mail: [email protected] Router Project: http://klamath.stanford.edu/or/
![Page 2: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/2.jpg)
2
Outline
Load-Balancing Background
Mis-sequencing Problem
Datapath Architecture First stage - Segmentation Second stage – Main Buffering Third stage - Reassembly
![Page 3: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/3.jpg)
3
Arbitration
160Gb/s
160Gb/s
SwitchFabric
• Line termination
• IP packet processing
• Packet buffering
• Line termination
• IP packet processing
• Packet buffering
160Gb/s
160Gb/s
Electronic
Linecard #1ElectronicLinecard #625
Request
Grant
(100Tb/s = 625 * 160Gb/s)
100Tb/s router
![Page 4: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/4.jpg)
4
Load-Balanced Switch
External Outputs
Internal Inputs
1
N
ExternalInputs
Load-balancing cyclic shift
Switching cyclic shift
1
N
1
N
11
2
2
![Page 5: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/5.jpg)
5
160 Gbps Linecard
Fixed-sizePackets
Reassembly
SegmentationLookup/
ProcessingR
1
N
2
VOQs
IntermediateInput Block
Load-balancing
Switching
Input Block
Output Block
R
R
R
R
R
![Page 6: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/6.jpg)
6
Outline
Load-Balancing Background
Mis-sequencing Problem
Datapath Architecture First stage - Segmentation Second stage – Main Buffering Third stage - Reassembly
![Page 7: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/7.jpg)
7
Problem: Unbounded Mis-sequencing
External Outputs
Internal Inputs
1
N
ExternalInputs
Spanning Set of Permutations
Spanning Set of Permutations
1
N
1
N
11
2
2
![Page 8: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/8.jpg)
8
Preventing Mis-sequencing
Uniform Frame Spreading: Group cells by frames of N cells each (frame building) Spread each frame across all middle linecards Each middle stage receives the same type of packets => has the same queue occupancy state
1 11
N
Middle stage
N N
1N1
N N
1N 1
![Page 9: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/9.jpg)
9
Outline
Load-Balancing Background
Missequencing Problem
Datapath Architecture First stage - Segmentation Second stage – Main Buffering Third stage - Reassembly
![Page 10: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/10.jpg)
10
Three stages on a linecard
Segmentation/Frame Building
1st stage
1
2
N
Main Buffering
2nd stage
1
2
N
R/N
RR R R
3rd stage
1
2
N
RR
Reassembly
![Page 11: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/11.jpg)
11
Technology Assumptions in 2005
•DRAM Technology•Access Time ~ 40 ns•Size ~ 1 Gbits•Memory Bandwidth ~ 16 Gbps (16 data pins)
•On-chip SRAM Technology•Access Time ~ 2.5 ns•Size ~ 64 Mbits
•Serial Link Technology•Bandwidth ~ 10 Gb/s•>100 serial links per chip
![Page 12: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/12.jpg)
12
First Stage
Segmentation
1
2
N
R
variable-sizepackets 128-byte
cells
16-bytes 1
2
N1
2
N
1
2
N
108-127
16-31
0-15
R/8
R/8
R/8
16-bytes
R/8
R/8
R/8
Frame Building
108-127
16-31
0-15
![Page 13: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/13.jpg)
13
Segmentation Chip (1st stage)
Segmentation
1
2
N
R
variable-sizepackets 128-byte
cells
R/8
R/8
R/8
Incoming: 16x10 Gb/s Outgoing: 8x2x10 Gb/s On-chip Memory: N x 1500 bytes = 7.2 Mbits 3.2ns SRAM
16-bytes
108-127
16-31
0-15
![Page 14: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/14.jpg)
14
Frame Building Chip (1st stage)
Incoming: 2x10 Gb/s Outgoing: 2x10 Gb/s On-chip Memory: N^2 x 16 bytes = 48 Mbits 3.2ns SRAM
16-bytes1
2
N
0-15
R/8
16-bytes
0-15
R/8
Frame Building
![Page 15: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/15.jpg)
15
Three stages on a linecard
Segmentation/Frame Building
1st stage
1
2
N
Main Buffering
2nd stage
1
2
N
R/N
RR R R
3rd stage
1
2
N
RR
Reassembly
![Page 16: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/16.jpg)
16
Packet Buffering Problem Packet buffers for a 160Gb/s router
linecard
BufferMemory
Write Rate, R
One 128B packetevery 6.4ns
Read Rate, R
One 128B packetevery 6.4ns
40Gbits
Buffer Manager
![Page 17: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/17.jpg)
17
Memory Technology
Use SRAM?+ Fast enough random access time, but- Too low density to store 40Gbits of data.
Use DRAM? + High density means we can store data, but- Can’t meet random access time.
![Page 18: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/18.jpg)
21
ArrivingPackets
R
Arbiter orScheduler
Requests
DepartingPackets
R
12
1
Q
21234
345
123456
Small head SRAM cache for FIFO heads
SRAM
Hybrid Memory HierarchyLarge DRAM memory holds the body of FIFOs
57 6810 9
79 81011
1214 1315
5052 515354
8688 878991 90
8284 838586
9294 9395 68 7911 10
1
Q
2
Writingb bytes
Readingb bytes
cache for FIFO tails
5556
9697
8788
57585960
899091
1
Q
2
Small tail SRAM
DRAM
![Page 19: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/19.jpg)
22
SRAM/DRAM results
How much SRAM buffering, given: DRAM Trc = 40ns Write and read a 128-byte cell every 6.4ns Let Q = 625, b = 2*40ns/6.4ns = 12.5
Two Options [Iyer] Zero Latency
Qb[2+lnQ] = 61k cells = 66 Mbits Some Latency
Q(b-1) = 7.5k cells = 7.5 Mbits
![Page 20: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/20.jpg)
23
Outline
Load-Balancing Background
Missequencing Problem
Datapath Architecture First stage - Segmentation Second stage – Main Buffering Third stage - Reassembly
![Page 21: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/21.jpg)
24
Problem Statement
QueueManager
40 Gb DRAM
160 Gb/s 160 Gb/s
160 Gb/s 160 Gb/sOne 128B cellevery 6.4ns
One 128B cellevery 6.4ns
Write Rate, R Read Rate, R
![Page 22: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/22.jpg)
25
Second Stage
R/8
R/8
R/8
Main Buffering
1
2
N
R/N
1
2
N
R/N
1
2
N
R/N
R/8
R/8
R/8
16-bytes
108-127
16-31
0-15
16-bytes
108-127
16-31
0-15
![Page 23: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/23.jpg)
26
Queue Manager Chip (2nd stage)
Incoming: 2x10 Gb/s Outgoing: 2x10 Gb/s 35 pins/DRAM x 5 DRAMs = 175 pins SRAM/DRAM Memory: Q(b-1) = 2.8 Mbits 3.2ns SRAM SRAM linked list = 1 Mbit 3.2ns SRAM
16-bytes
0-15
R/8
16-bytes
0-15
R/8
Main Buffering
1
2
N
R/N
5 x 1Gb DRAM
R/4 R/4
![Page 24: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/24.jpg)
27
Outline
Load-Balancing Background
Missequencing Problem
Datapath Architecture First stage - Segmentation Second stage – Main Buffering Third stage - Reassembly
![Page 25: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/25.jpg)
28
Three stages on a linecard
Segmentation/Frame Building
1st stage
1
2
N
Main Buffering
2nd stage
1
2
N
R/N
RR R R
3rd stage
1
2
N
RR
Reassembly
![Page 26: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/26.jpg)
29
Third stage
Reassembly
1
2
N
R
variable-sizepackets
R/8
R/8
R/8
Incoming: 8x2x10 Gb/s Outgoing: 16x10 Gb/s On-chip Memory: N x 1500 bytes = 7.2 Mbits 3.2ns
SRAM
16-bytes
108-127
16-31
0-15
![Page 27: 1 OR Project Group II: Packet Buffer Proposal Da Chuang, Isaac Keslassy, Sundar Iyer, Greg Watson, Nick McKeown, Mark Horowitz E-mail: stchuang@stanford.edu](https://reader031.vdocuments.net/reader031/viewer/2022032309/56649d4a5503460f94a26474/html5/thumbnails/27.jpg)
30
1st stage 1 segmentation chip 8 frame building chips
2nd stage 8 queue manager chips 40 1 Gb DRAMs
3rd stage 1 reassembly chip
Total chip count 18 ASIC chips 40 1 Gb DRAMs
Linecard Datapath Requirements