lhcone status and future
DESCRIPTION
LHCONE status and future. Alice workshop Tsukuba, 7 th March 2014 [email protected]. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t. Summary. - Networking for WLCG - LHCOPN - LHCONE - services - how to join - LHCONE in Asia. Networking for WLCG. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/1.jpg)
1
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
LHCONE status and future
Alice workshopTsukuba, 7th March 2014
![Page 2: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/2.jpg)
2
Summary
- Networking for WLCG
- LHCOPN
- LHCONE
- services
- how to join
- LHCONE in Asia
![Page 3: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/3.jpg)
3
Networking for WLCG
![Page 4: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/4.jpg)
4
Worldwide LHC Computing Grid
WLCG sites:- 1 Tier0 (CERN)
- 13 Tier1s
- ~170 Tier2s
- >300 Tier3s worldwide
![Page 5: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/5.jpg)
5
Planning for Run2
“The Network infrastructure is the most reliable service we have”
“Network Bandwidth (rather than disk) will need to scale more with
users and data volume”
“Data placement will be driven by demand for analysis and not pre-
placement”
Ian Bird, WLCG project leader
![Page 6: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/6.jpg)
6
Computing model evolution
Original MONARCH model
Model evolution
![Page 7: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/7.jpg)
7
Technology Trends
- Commodity Servers with 10G NICs- High-end Servers with 40G NICs- 40G and 100G interfaces on switches and routers
Needs for 100Gbps backbones to host large data flows >10Gbps and soon
>40Gbps
![Page 8: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/8.jpg)
8
Role of Networks in WLCG
Computer Networks even more essential component of WLCG
Data analysis in Run 2 will need more network bandwidth between any pair of
sites
![Page 9: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/9.jpg)
9
LHCOPNLHC Optical Private Network
![Page 10: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/10.jpg)
10
What LHCOPN is:
Private network connecting Tier0 and Tier1s
Reserved to LHC data transfers and analysis
Dedicated large bandwidth links
Highly resilient architecture
![Page 11: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/11.jpg)
11
A collaborative effort
Layer3: Designed, built and operated by the Tier0-Tier1s community
Layer1-2: Links provided by Research and Education network providers: Asnet, ASGCnet, Canarie, DFN, Esnet, GARR, Geant, JANET, Kreonet, Nordunet, Rediris, Renater, Surfnet, SWITCH, TWAREN, USLHCnet
![Page 12: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/12.jpg)
12
Topology
█ = Alice █ = Atlas █ = CMS █ = [email protected] 20131113
TW-ASGC█ █
CA-TRIUMF█
US-T1-BNL█
US-FNAL-CMS█
FR-CCIN2P3█ █ █ █
ES-PIC█ █ █
IT-INFN-CNAF█ █ █ █
DE-KIT█ █ █ █
NL-T1█ █ █
UK-T1-RAL █ █ █ █
NDGF█ █
RRC-K1-T1█ █ █
KR_KISTI█
CH-CERN█ █ █ █
![Page 13: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/13.jpg)
13
Technology
- Single and bundled long distance 10G Ethernet links
- Multiple redundant paths. Star and Partial-Mesh topology
- BGP routing: communities for traffic engineering, load balancing.
- Security: only declared IP prefixes can exchange traffic.
![Page 14: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/14.jpg)
14
LHCOPN future
- The LHCOPN will be kept as the main network to exchange data among the Tier0 and Tier1s
- Links to the Tier0 may be soon upgraded to multiple 10Gbps (waiting for Run2 to see the real needs)
![Page 15: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/15.jpg)
15
LHCONELHC Open Network Environment
![Page 16: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/16.jpg)
16
New computing model impact
- Better and more dynamic use of storage
- Reduced load on the Tier1s for data serving
- Increased speed to populate analysis facilities
Needs for a faster, predictable, pervasive network connecting
Tier1s and Tier2s
![Page 17: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/17.jpg)
17
Requirements from the Experiments
- Connecting any pair of sites, regardless of the continent they reside
- Site's bandwidth ranging from 1Gbps (Minimal), 10Gbps (Nominal), to 100G (Leadership)
- Scalability: sites are expected to grow
- Flexibility: sites may join and leave at any time
- Predictable cost: well defined cost, and not too high
![Page 18: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/18.jpg)
18
LHCONE concepts
- Serving any LHC sites according to their needs and allowing them to grow
- Sharing the cost and use of expensive resources
- A collaborative effort among Research & Education Network Providers
- Traffic separation: no clash with other data transfer, resource allocated for and funded by the HEP community
![Page 19: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/19.jpg)
19
Governance
LHCONE is a community effort.
All stakeholders involved: TierXs, Network Operators, LHC Experiments, CERN.
![Page 20: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/20.jpg)
20
LHCONE services
L3VPN (VRF): routed Virtual Private Network - operational
P2P: dedicated, bandwidth guaranteed, point-to-point links - development
perfSONAR: monitoring infrastructure
![Page 21: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/21.jpg)
21
LHCONE L3VPN
![Page 22: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/22.jpg)
22
What LHCONE L3VPN is:
Layer3 (routed) Virtual Private Network
Dedicated worldwide backbone connecting Tier1s, T2s and T3s at high bandwidth
Reserved to LHC data transfers and analysis
![Page 23: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/23.jpg)
23
Advantages
Bandwidth dedicated to LHC data analysis, no contention with other research projects
Well defined cost tag for WLCG networking
Trusted traffic that can bypass firewalls
![Page 24: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/24.jpg)
24
LHCONE L3VPN architecture- TierX sites connected to National-VRFs or Continental-VRFs- National-VRFs interconnected via Continental-VRFs - Continental-VRFs interconnected by trans-continental/trans-oceanic links
Acronyms: VRF = Virtual Routing Forwarding (virtual routing instance)
ContinentalVRFs
ContinentalVRFs
ContinentalVRFs
ContinentalVRFs
NationalVRFs
TierXs
NationalVRFs
TierXs
NationalVRFs
TierXsTierXs
LHCONE
NationalVRFs
TierXs
![Page 25: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/25.jpg)
25
Current L3VPN topology
LHCONE VRF domain
End sites – LHC Tier 2 or Tier 3 unless indicated as Tier 1
Regional R&E communication nexus
Data communication links, 10, 20, and 30 Gb/s
See http://lhcone.net for details.
NTU
Chicago
credits: Joe Metzger, ESnet
![Page 26: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/26.jpg)
26
Status
Over 15 national and international Research Networks
Several Open Exchange Points including NetherLight, StarLight, MANLAN, CERNlight and others
Trans-Atlantic connectivity provided by ACE, GEANT, NORDUNET and USLHCNET
~50 end sites connected to LHCONE:- 8 Tier1s- 40 Tier2s
Credits: Mian Usman, DanteMore Information: https://indico.cern.ch/event/269840/contribution/4/material/slides/0.ppt
![Page 27: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/27.jpg)
27
Operations
Usual Service Provider operational model: a TierX must refer to the VRF providing the local connectivity
Bi-weekly call among all the VRF operators and concerned TierXs
![Page 28: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/28.jpg)
28
How to join the L3VPN
![Page 29: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/29.jpg)
29
Pre-requisites
The TierX site needs to have:
- Public IP addresses
- A public Autonomous System (AS) number
- A BGP capable router
![Page 30: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/30.jpg)
30
How to connect
The TierX has to:
- Contact the Network Provider that runs the closest LHCONE VRF
- Agree on the cost of the access
- Lease a link from the TierX site to the closest LHCONE VRF PoP (Point of Presence)
- Configure the BGP peering with the Network Provider
![Page 31: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/31.jpg)
31
TierX routing setup
- The TierX announce only the IP subnets used for WLCG servers
- The TierX accepts all the prefixes announced by the LHCONE VRF
- The TierX must assure traffic symmetry: injects only packets sourced by the announced subnets
- LHCONE traffic may be allowed to bypass the central firewall (up to the TierX to decide)
![Page 32: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/32.jpg)
32
Symmetric traffic is essential
Beware: statefull firewalls discard unidirectional TCP connections
CERNLCG
backbone
CERNCampus
backboneInternet
LHCONE
Default
Defa
ult
TierX LCG destinations
All destinations
All CERN's destinations
BorderNetwork
LCG host
Campus host
LCG host
Stateful firewallDrops asymmetric TCP
flows
Stateless ACLs
Campus host
LHCONE host to LHCONE host
CERN's LHCONE host to TierX not LHCONE host
CERN's not LHCONE host to TierX's LHCONE host
Default
CERN
CERN LCG
destination
s
All destinations
All destinations
CERN LCG destinations
TierX
![Page 33: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/33.jpg)
33
Symmetry setup
To achieve symmetry, a TierX can use one of the following techniques:
- Policy Base Routing (source-destination routing)
- Physically Separated networks
- Virtually separated networks (VRF)
- Scienze DMZ
![Page 34: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/34.jpg)
34
Scienze DMZ
http://fasterdata.es.net/science-dmz/science-dmz-architecture/
![Page 35: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/35.jpg)
35
LHCONE P2PGuaranteed bandwidth point-to-point links
![Page 36: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/36.jpg)
36
What LHCONE P2P is (will be):
On demand point-to-point (P2P) link system over a multi-domain network Provides P2P links between any pair of TierX
Provides dedicated P2P links with guaranteed bandwidth (protected from any other traffic)
Accessible and configurable via software API
![Page 37: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/37.jpg)
37
Status
Work in progress: still in design phase
Challenges:
- multi-domain provisioning system
- intra-TierX connectivity
- TierX-TierY routing
- interfaces with WLCG software
![Page 38: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/38.jpg)
38
LHCONE perfSONAR
![Page 39: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/39.jpg)
39
What LHCONE perfSONAR is
LHCONE Network monitoring infrastructure
Probe installed at the VRFs interconnecting points and at the TierXs
Accessible to any TierX for network healthiness checks
![Page 40: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/40.jpg)
40
perfSONAR
- framework for active and passive network probing
- developed by Internet2, Esnet, Geant and others
- two interoperable flavors: perfSONAR-PS and perfSONAR-MDM
- WLCG recommended version: perfSONAR-PS
![Page 41: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/41.jpg)
41
Status
Endorsed by WLCG to be a standard WLCG service
Probes already deployed in many TierXs.
Being deployed in the VRF networks
Full information:https://twiki.cern.ch/twiki/bin/view/LCG/PerfsonarDeployment
![Page 42: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/42.jpg)
42
LHCONE-L3VPN in Asia
![Page 43: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/43.jpg)
43
Connectivity status
Only few sites connected to LHCONE-L3VPN in ASIA via ASGC or with direct link to the US or Europe
Connectivity between ASIA and North America not scarce, but transit to Europe may not be adequate
Un-coordinated effort
![Page 44: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/44.jpg)
44
Existing connectivity
10G
10G
Chicago
TAIWAN
HongKong
Amsterdam
Geneva
2.5G2.5GTokyo
10G
10G
2.5G
LA
Palo Alto New York2.5G5G
2.5G
TANet
TWARENASNet
ASGCNet
2.5G
5G
2.5G
622M
2.5G
2.5G
SeattleDaejeon
15G
15G
KREONet2
Disclaimer: list of links not exhaustiveCredits: Hsin Yen Chen
10G
![Page 45: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/45.jpg)
45
Working together
ASCG is willing to share the use of their links to North-America and Europe with other Asian TierXs
Anyone interested to connect to the Asian LHCONE or share their trans-continental links, please get in touch with us
![Page 46: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/46.jpg)
46
Anyway: You have to tune!
TCP Throughput <= TCPWinSize / RTT
Tokyo-CERN RTT (Round Trip Time): 280 msDefault Max TCPWinSize for Linux = 256KBytes ( = 2.048Mbit)
Tokyo-CERN throughput <= 2.048Mb / 0.280sec = 7.31Mbps :-(
Remote TierXs must tune server and client TCP Kernel parameters to get
decent throughput!
![Page 47: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/47.jpg)
47
LHCONE evolution
![Page 48: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/48.jpg)
48
LHCONE evolution
- VRFs have started upgrading internal links and links to TierXs to 100Gbps
- VRFs interconnecting links will be upgraded to 100Gbps. 100Gbps Transatlantic link being tested.
- Operations need to be improved, especially how to support a TierX in case of performance issue
- perfSONAR deployment will be boosted
![Page 49: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/49.jpg)
49
LHCONE evolution
- LHCONE-P2P take off still uncertain
- LHCONE-L3VPN must be better developed in ASIA
![Page 50: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/50.jpg)
50
Conclusions
![Page 51: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/51.jpg)
51
Conclusions
- New Computing Models will relay even more on good and abundant network connectivity
- TierXs need to improve their network connectivity
- LHCONE-L3VPN is a viable solution already adopted by many Tier1/2s
![Page 52: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/52.jpg)
52
More information
Last LHCONE workshop:https://indico.cern.ch/event/289679/
LHCONE websites:http://lhcone.nethttps://twiki.cern.ch/twiki/bin/view/LHCONE/WebHome
Weekly audio conference: Monday 14:30 GMT, alternating every second week architecture and operations
Mailing lists:[email protected]@cern.ch
![Page 53: LHCONE status and future](https://reader036.vdocuments.net/reader036/viewer/2022062500/56814fc3550346895dbd8015/html5/thumbnails/53.jpg)
53
Questions?