experiences from slac sc2004 bandwidth challenge
DESCRIPTION
Experiences from SLAC SC2004 Bandwidth Challenge. Les Cottrell, SLAC www.slac.stanford.edu/grp/scs/net/talk03/bwc-04.ppt. Sponsors/partners. SLAC/SC04 Bandwidth Challenge (plan C). PSC SciNet. 2 Sun Opteron/Chelsio-10GE. SLAC/FNAL booth (2418). 6 Boston file servers 1 GE. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Experiences from SLAC SC2004 Bandwidth Challenge](https://reader033.vdocuments.net/reader033/viewer/2022051821/56815884550346895dc5e552/html5/thumbnails/1.jpg)
1
Experiences from SLAC SC2004 Bandwidth Challenge
Les Cottrell, SLACwww.slac.stanford.edu/grp/scs/net/talk03/bwc-04.ppt
![Page 2: Experiences from SLAC SC2004 Bandwidth Challenge](https://reader033.vdocuments.net/reader033/viewer/2022051821/56815884550346895dc5e552/html5/thumbnails/2.jpg)
2
Sponsors/partners
![Page 3: Experiences from SLAC SC2004 Bandwidth Challenge](https://reader033.vdocuments.net/reader033/viewer/2022051821/56815884550346895dc5e552/html5/thumbnails/3.jpg)
3
SLAC/SC04 Bandwidth Challenge (plan C)
ESnet/QWest OC192/SONET
Sunnyvale/Qwest/ESnet
Juniper T320
Sunnyvale/Level(3)/NLR
SLAC/FNAL booth (2418)
1400 Kifer
1380 Kifer
10Gbps from NLR (via SEA, DEN, CHI)
PSC SciNet
SiSi
LoanedCisco Rtr
NLR-PITT-SUNN-10GE-17 NLR-PITT-SUNN-10GE-17
1 Sun file server 1 GE
2 Sun Opteron/Chelsio-10GE
SiSi
1554015808
NLR demarc 15808
15454
NLR demarc
SLACCisco Rtr
6 Sun Opteron/Chelsio-10GE
6 Boston file servers 1 GE 2 Sun file server 1 GE2 Sun Opteron/S2io-10GE
1 Sun Opteron/Chelsio-10GE
![Page 4: Experiences from SLAC SC2004 Bandwidth Challenge](https://reader033.vdocuments.net/reader033/viewer/2022051821/56815884550346895dc5e552/html5/thumbnails/4.jpg)
4
SC2004: Tenth of a Terabit/s Challenge
Joint Caltech, SLAC, Joint Caltech, SLAC, FNAL, CERN, UF, FNAL, CERN, UF, SDSC, BR, KR, ….SDSC, BR, KR, ….
10 10 Gbps waves to 10 10 Gbps waves to
HEP on show floorHEP on show floor Bandwidth Bandwidth
challenge: aggregate challenge: aggregate throughput throughput of 101.13 Gbpsof 101.13 Gbps
FAST TCPFAST TCP
![Page 5: Experiences from SLAC SC2004 Bandwidth Challenge](https://reader033.vdocuments.net/reader033/viewer/2022051821/56815884550346895dc5e552/html5/thumbnails/5.jpg)
5
Components10 Gbps NICs
S2io
Chelsio
1982 10Mbps3COM
v20z
v40z
3510 disk array
SR XENPAK
SVL/NLR
![Page 6: Experiences from SLAC SC2004 Bandwidth Challenge](https://reader033.vdocuments.net/reader033/viewer/2022051821/56815884550346895dc5e552/html5/thumbnails/6.jpg)
6
Challenge aggregates from SciNet
• Aggregate Caltech & SLAC booth, in & out
• 7 lambdas to Caltech, 3 ro SLAC
![Page 7: Experiences from SLAC SC2004 Bandwidth Challenge](https://reader033.vdocuments.net/reader033/viewer/2022051821/56815884550346895dc5e552/html5/thumbnails/7.jpg)
7
Challenge aggregates from MonALISA
• Sustained ~ 10Gbps for extended periods
![Page 8: Experiences from SLAC SC2004 Bandwidth Challenge](https://reader033.vdocuments.net/reader033/viewer/2022051821/56815884550346895dc5e552/html5/thumbnails/8.jpg)
8
Weathermap showing 8.7Gbps on ESnet
![Page 9: Experiences from SLAC SC2004 Bandwidth Challenge](https://reader033.vdocuments.net/reader033/viewer/2022051821/56815884550346895dc5e552/html5/thumbnails/9.jpg)
9
To/From SLAC booth• NLR: 9.43Gbps (9.07 goodput) + 5.65Gbps
(5.44Gbps goodput) in reverse– Two hosts to two hosts
• ESnet: 7.72Gbps (7.43Gbps goodput)– Only one 10Gbps host at SVL
• Single V40Z host with 2*10GE NICs to 2*V20Z across country got 11.4Gbps
• S2io and Chelsio (& Cisco & Juniper) all interwork
• Chelsio worked stably on uncongested paths
![Page 10: Experiences from SLAC SC2004 Bandwidth Challenge](https://reader033.vdocuments.net/reader033/viewer/2022051821/56815884550346895dc5e552/html5/thumbnails/10.jpg)
10
TOE• Chelsio had TCP Offload Engine
– Utilization factor of throughput & parallel streams
– Reduced cpu c.f. S2io non0TOE by factor ~ 3
![Page 11: Experiences from SLAC SC2004 Bandwidth Challenge](https://reader033.vdocuments.net/reader033/viewer/2022051821/56815884550346895dc5e552/html5/thumbnails/11.jpg)
11
Challenges• Could not get 10Gbps waves to SLAC only SVL
– Equipment in 3 locations• Keeping configs in lock-step (no NFS, no name
service)• Security concerns, used iptables• Machines only available 2 weeks before, some not
until we got to SC04• Jumbo frames not configured correctly at SLAC booth,
used 1500B frames mainly• Mix of hdw/swr: Opterons with various GHz & disks,
Xeons; Solaris 10, Linux 2.4, 2.6• Coordination between booths (sep by 100 yds)• Everything state of art (Linux 2.6.6, SR XENPAKs,
NICs
![Page 12: Experiences from SLAC SC2004 Bandwidth Challenge](https://reader033.vdocuments.net/reader033/viewer/2022051821/56815884550346895dc5e552/html5/thumbnails/12.jpg)
12
Award