esnet update · 2016-09-13 · • pilot underway: slac • looking at: – network architecture...
TRANSCRIPT
Supporting Advanced Scientific Computing Research • Basic Energy Sciences • Biological and Environmental Research • Fusion Energy
Sciences • High Energy Physics • Nuclear Physics
ESnetUpdate
Feb3,2009ESCC,SaltLakeCity
SteveCo6er,DeptHead
LawrenceBerkeleyNaDonalLab
Outline
• StaffUpdates• NetworkUpdate• AdvancedNetworkingIniDaDve• ESnetProjects• InfrastructureProjects• StaffProjects
StaffUpdate
Newhires:
• HingChow:ProjectManager(ANI)• ChrisTracy:Network/SoVwareEngineer(ANI)• AndyLake:SoVwareEngineer(ANI)• InderMonga:Network/SoVwareEngineer(ANI)• JosefGrosch:SysAdmin• PosiDonsposted:
– ChiefInformaDonStrategist– SoVwareEngineer
NetworkUpdate
CurrentStatus,Upgrades
ESnet4Network
5
EquipmentUpgrades/InstallsPeeringupgrades:
• EQX‐SJ:installedMX480onOct15th
• EQX‐ASH:installedMX480onNov30th
• EQX‐CHI:PendingMX480installonFeb18th
Site/hubupgrades:• ORNLM20:UpgradedtoMX480onDec17th
• AMESLABM10:UpgradedtoM10ionJan13th
• FORR7206:UpgradedtoM7ionDec22nd
• DOE‐GTN7206:ScheduledupgradetoM10ilateFeb.
• DOE‐NNSAM10:ScheduledupgradetoM7ilateFeb.
• PPPLM120:ScheduledupgradetoMX480mid‐Mar/Apr
• GAM7i:ScheduledupgradedtoMX480mid‐Mar/Apr
CircuitInstalls• 10GconnecDonatBOISwithPNNLforbackuppeeringNov10th• 10GpeeringatPNWG‐HUBwithKorea(KSTAR&KISTI)Nov11th
• CombineofLOSA‐SUNN&ELPA‐LOSAintonewELPA‐SUNNSDN(priortothedecommissionofLOSA‐HUB)Dec.3rd
• 10GEquinixASH(DC2)fabricupgradedonJan14th
• 10GEquinixSJ(SV1)fabricupgradedonJan19th
• OC12betweenDENV‐HUBandPantexJan28th
• DS3betweenWASH‐HUBandSNL‐DC(nowintest)
• 10GEQX‐CHIfabricupgradescheduledFeb19th• 1GEwaveinBOIStoINLviaIRON(TBD)• 1GElinksinD.C.AreaforGermantown,INtoWASH‐HUB(ordered)
• OC3cforNSOtoLASV‐HUB(ordered)• 10GWavebetweenCHIC‐HUBandEQX‐CHI(ordered)
• 10GWavebetweenWASH‐HUBandEQX‐ASH(pending)
• Future10GpeeringwithMERIT@Starlight
• FutureaddiDonal10GpeeringwithGPN@KANS‐HUB
PlannedUpgrades/Installs
ESnetTraffic
Jan, 1990 A
pr, 1990 Jul, 1990 O
ct, 1990 Jan, 1991 A
pr, 1991 Jul, 1991 O
ct, 1991 Jan, 1992 A
pr, 1992 Jul, 1992 O
ct, 1992 Jan, 1993 A
pr, 1993 Jul, 1993 O
ct, 1993 Jan, 1994 A
pr, 1994 Jul, 1994 O
ct, 1994 Jan, 1995 A
pr, 1995 Jul, 1995 O
ct, 1995 Jan, 1996 A
pr, 1996 Jul, 1996 O
ct, 1996 Jan, 1997 A
pr, 1997 Jul, 1997 O
ct, 1997 Jan, 1998 A
pr, 1998 Jul, 1998 O
ct, 1998 Jan, 1999 A
pr, 1999 Jul, 1999 O
ct, 1999 Jan, 2000 A
pr, 2000 Jul, 2000 O
ct, 2000 Jan, 2001 A
pr, 2001 Jul, 2001 O
ct, 2001 Jan, 2002 A
pr, 2002 Jul, 2002 O
ct, 2002 Jan, 2003 A
pr, 2003 Jul, 2003 O
ct, 2003 Jan, 2004 A
pr, 2004 Jul, 2004 O
ct, 2004 Jan, 2005 A
pr, 2005 Jul, 2005 O
ct, 2005 Jan, 2006 A
pr, 2006 Jul, 2006 O
ct, 2006 Jan, 2007 A
pr, 2007 Jul, 2007 O
ct, 2007 Jan, 2008 A
pr, 2008 Jul, 2008 O
ct, 2008 Jan, 2009 A
pr, 2009 Jul, 2009 O
ct, 2009 Jan, 2010 A
pr, 2010 Jul, 2010 O
ct, 2010
0.0
0.1
1.0
10.0
100.0
1000.0
10000.0
100000.0
Actual
Exponential regression extended 12 months beyond actualESnet Accepted Traffic (TB/mo) - Log Scale
Projected volume for Dec, 2010: 10442 TB
Actual volume for Dec, 2009: 3562 TB
Newtrend?
MonitoringtrafficonprimaryandsecondaryESnet/USLHCnetinterconnectsatStarLight
8Gprimary
6GSecondary
Dec12‐2.36TeVLHCRun
MonitoringtrafficonprimaryandsecondaryESnet/USLHCnetinterconnectsatStarLight
FermiTraffic:Dec13–Feb2
AdvancedNetworkingIniDaDve
PrototypeNetworkandTestbed
• ANIProjectscope:– Buildend‐to‐end100GbpsprototypenetworkbetweenDOEsupercomputersandMANLAN
– Buildanetworktestbedfacilityforresearchersandindustry• DOEhasfundedanaddiDonal$5Minnetworkresearchthatwillusethetestbedfacility
• Magellan:– SeparateDOE‐fundednaDonwidescienDficmid‐rangedistributedcompuDnganddataanalysistestbedtoexplorewhethercloudcompuDngcanhelpmeettheoverwhelmingdemandforscienDficcompuDng
– NERSC/LBNL&ALCF/ANLconfiguredwithmulDple10’softeraflopsandmulDplepetabytesofstorage,aswellasappropriatecloudsoVware
AdvancedNetworkingIniDaDve
13
• Prototypenetwork:– Acceleratethedeploymentof100Gbpstechnologies– BuildapersistentinfrastructurethatwilltransiDontotheproducDonnetwork~2012
• KeysteptowardDOE’svisionofa1‐TerabitnetworklinkingDOEsupercompuDngcentersandexperimentalfaciliDes
• Testbed:– BuildanexperimentalnetworkresearchenvironmentatsufficientscaletousefullytestexperimentalapproachestonextgeneraDonnetworks
• Fundedfor3years,thenrollintotheESnetprogram• Breakable,reserveable,configurable,rese6able• EnableR&Datspeedsupto100Gbps
ANIProjectGoals
14
ANITopology
15
Magellan
Magellan
OpDcalclientsideconnecDons,i.e.1x100GE
man‐wdm
BayExpressBackBone
bb‐wdm
NERSC
pe‐wdm
ce‐rtr
1x100GE
3x100GE
man‐wdm
ChiExpressBackBone
bb‐wdm
pe‐rtr1x100GE
ANL
pe‐wdm
ce‐rtr
1x100GE
ORNL
pe‐wdm
ce‐rtr
1x100GE
OpDcalclientsideconnecDons,i.e.1x100GE
bb‐wdmAofA
aofa‐wdm
newy‐wdm
bnl‐wdm1 bnl‐wdm2
LIMAN
1x100GE
StarlightPeeringExchange
1x100GE
MANLANPeeringExchange
1x100GE
ANIBaselineDesignTestbed
• Progression:– Startoutasatabletoptestbed,thenmoveouttothewide‐areawhen
100Gbpsavailable• CapabiliDes:
– Abilitytosupportend‐to‐endnetworking,middlewareandapplicaDonexperiments,includinginteroperabilitytesDngofmulD‐vendor100Gbpsnetworkcomponents
– Dynamicnetworkprovisioning– PlantoacquiredarkfiberonaporDonoftestbedfootprinttoenable
hybrid(layer0‐3)networkresearch– UseVirtualMachinetechnologytosupportprotocolandmiddleware
research– Detailedmonitoringsoresearcherswillhaveaccesstoallpossible
monitoringdatafromthenetworkdevices
TestbedOverview
17
north‐wdm1
10GE
south‐wdm1
Prod.
north‐wdm2
1GE
east‐wdm1
east‐wdm2
10GE
10GE
1GE
10GE
10GE
1GE
1GE
2x10GE
OpenflowSwitch
10GTester
FS/BS/Apphost
MonitoringHost
OpenflowSwitch
south‐wdm2
1GE
1GE
1GE
10GTesterFS/BS/Apphost
MonitoringHost
10GTester
10GTester
WDMLink10GELink1GELink
TabletopTestbedDesign
• Tabletoptestbedequipment–ordered,rackedatLBL• 100Gbpstechnologyresearch&evaluaDonphase‐ongoing
– MeeDngs/briefingswithvendors– EquipmentinESnetlab
• TransportRFPwri6en–goingthroughreviews– Acquire100Gbpswaveservicefromacarrier– Don’tneedtoown/controlopDcalgear– PlantorunOSCARSlayer2/3servicesacrossnetwork– DarkfiberispartofDOE’slong‐termresearchagenda
• RouDng/SwitchRFP–Summer2010– ESnetwillpurchasethisequipment– WillconducttesDng/evaluaDonaspartofselecDonprocess
ANIProgresstoDate
19
ESnetProjects
perfSONAR,OSCARS,Fenius,etc.
perfSONAR
• ESnetisakeymemberoftheperfSONARcollaboraDon–h6p://www.perfsonar.net
• Numeroustesthostsdeployed,automatedtestsarerunregularly(h6p://stats1.es.net)
• TesthostsareavailabletoESnetsitesandR&Ecollaboratorsforbwctl/iperftests
• TestandmeasurementisveryhelpfulinlocaDngthecauseofnetworkperformanceproblems
OSCARS:“MulD‐Domain,VirtualCircuits”asaService
• SuccessfullydeployedwithinESnetSDN• OSCARSSoVwareisOpen‐source(oscars‐idc.googlecode.com)
– Aresourceforthecommunity– Example:Internet2IONleveragesOSCARS
• Ongoingchallenge:Builddual‐purposesoVware– Enableresearcherstoinnovateusingthisframework– Providerobustproduct‐gradesoVware– TakeadvantageofnewinnovaDonsandresearchinthisfield
• DirecDonforward:BuildcriDcalmassaroundtheopen‐sourceeffort• Collaboratewithlike‐mindedresearchersandopen‐source
projectslikeOpen‐DRACandOpenflow
ESnetIP&OSCARSTraffic
Jan
, 20
00
Ap
r, 20
00
Jul, 2
00
0
Oct, 2
00
0
Jan
, 20
01
Ap
r, 20
01
Jul, 2
00
1
Oct, 2
00
1
Jan
, 20
02
Ap
r, 20
02
Jul, 2
00
2
Oct, 2
00
2
Jan
, 20
03
Ap
r, 20
03
Jul, 2
00
3
Oct, 2
00
3
Jan
, 20
04
Ap
r, 20
04
Jul, 2
00
4
Oct, 2
00
4
Jan
, 20
05
Ap
r, 20
05
Jul, 2
00
5
Oct, 2
00
5
Jan
, 20
06
Ap
r, 20
06
Jul, 2
00
6
Oct, 2
00
6
Jan
, 20
07
Ap
r, 20
07
Jul, 2
00
7
Oct, 2
00
7
Jan
, 20
08
Ap
r, 20
08
Jul, 2
00
8
Oct, 2
00
8
Jan
, 20
09
Ap
r, 20
09
Jul, 2
00
9
Oct, 2
00
9
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Accepted
OSCARS AcceptedESnet Accepted Traffic (TB/mo)
NoBficaBonBroker• ManageSubscripDons• ForwardNoDficaDons
AuthN• AuthenDcaDon
PathSetup• NetworkElement
Interface
Coordinator• WorkflowCoordinator
PCE• ConstrainedPathComputaDons
TopologyBridge• TopologyInformaDon
Management
WSAPI• ManagesExternalWS
CommunicaDons
ResourceManager• ManageReservaDons
• AudiDng
Lookup• Lookupservice
AuthZ*• AuthorizaDon
• CosDng
*Dis%nct Data and Control Plane Func%ons
WebBrowserUserInterface
50%
80%
50%
95%
50%
95%
20%
50%70%
90%
60%
OSCARS0.6–TargetRelease3/10
Fenius
• Atthe9thAnnualGlobalLambdaGridWorkshopinDaejeon,Korea.– ESnet,KISTI,AISTandtheEU‐fundedPhosphorusprojectsuccessfullydemonstratedinteroperabilitybetweentheirnetworkresourceschedulingsystems
• CoordinatedwithintheacDviDesoftheGLIFconsorDumGNIAPITaskForce– DevelopedspecializedsoVwaretoenablethedifferentnetworkschedulingservicestobeusedandmonitoredthroughonecommoninterface.
• DemonstratedagainatSupercompuDng2009
SiteOutreachProgram
• StartedJan1,ledbyEliDart• GoalistoincreaseeffecDveuseofnetworksforscience– LeverageESnet’sexperienceinhelpingsitessolveproblemsandincreaseperformance
– Understandsitenetworkinfrastructure,drivers,andlong‐termplans
– Helpsitesanddisciplinesbuildnetworkswell‐matchedtotheirneeds
SiteOutreachProgram
• Pilotunderway:SLAC• Lookingat:
– Networkarchitecture• Impactof“convergednetworks”onhighbandwidthdatatransfers–andpossibleneedforseparaDonofScienceandEnterprisenetworks
• Adequatebufferingonswitchesandrouters– HostandsystemconfiguraDon
• Dedicatedhostsforwideareadatatransfer• ProperTCPtuning
– Testandmeasurementinfrastructure(e.g.perfSONAR)
LongIslandMAN
RFPresponsesdue:Feb22nd
• ReplicaDon&geographicdispersionofkeymanagement– netHSM(networkenabledHardwareSecurityModule)areusedtomanagethekeypairsthattheCAuses
• ReplicaDon&geographicdispersionofCA(signinginiDator,UI,database)– RedHatCerDficateSystem
• ReplicaDon&geographicdispersionofCRLpublishing– ANYCAST– CRLDistribuDonfromtheCloud–egAmazonCloudfront
• Remoteoperator&geographicdispersionofoperator– Remoteoperatorservice
• Nagiosmonitoringsystem– Anewdevelopmentforthisproject
ProvideHighAvailabilityCA/PKIService
DOEGridsCAwithHighAvailability
Eastcoast
RemoteOperatorRemoteOperator
DOEGridsCAmaster DOEGridsCAclone
netHSM
CRLdelivery
LDAP LDAP
Westcoast
...
......
...
netHSM
MidWest?
" "
"
Available"
ScienceIdenDtyFederaDon
• InteroperableIdenDtyforDOElabs… based on the well‐known
• ShibbolethauthenDcaDon&authorizaDonsoVware
… so that labs can
• FederatewithInCommon– WhichisUSHigherEducaDonShibbolethFederaDon:seeincommonfederaDon.org
– AndotherfederaDonsasneeded
ScienceIdenDtyFederaDonProgram• Training–getearlyadoptersuptospeedonShibbolethIDPand
integraDonwiththeirhomeservice– ShibfestMar30‐31atFNAL–detailstocomesoon
• WriteaminimalcharterforacDviDes• ApplicaDonsandserviceswilldefinethemissionulDmately– NeedtolookatDOEservices,userfaciliDes,supercomputercenters&cfor
goodintegraDoncandidates– Lookatspecificlabneedsandinterests–expressinga6ributeslike“LoA”or
“ciDzenship”• ProvidedemonstraDonservices
– Confluence(aCMSwithwiki‐likefeatures)– AdemoGridcredenDalCA
• AcDviDestocome:– ExploringinteroperaDonwithotherservices
• CILogon,otherSAMLacDviDes,EUfederaDons– AlternaDvetechnology:OAuth,OpenID– TFPAP&ICAM–alternaDvestofederaDon
ScienceIdenDtyFederaDonContact
• Ifyou’reinterested,andhavesomerelaDonshiptotheDOElabcommunityorprojects:h6p://groups.google.com/group/science‐federaDon
We’reusingthis“privatepublic”grouptobootstrap;
InfrastructureProjects
OpenDevNet,DNSSEC,Spectrum,etc.
ESnetOpenDevNetTesDngtool:• SoVwarelifecycle:build,debug,QA
– PlaxormtobuildsoVware‐as‐service,readyfordeploymentonproducDonservers
• ‘Slice’model:madeofvirtualresources(VM,virtualnetworktopologies)andphysicalresources(routers,performancenodes,WANaccess)
• TesDngplaxormforthirdpartytechnologies:sandbox,demo,tesDng.
• Deploymentofservicesrequiredforexperiments• Deploymentofuser‐specifiednetworktopologies• TesDngplaxormforothertestbeds
Contact:[email protected]
OpenDevNetArchitecture
DNSSEC
DomainNameSystemSecurityExtensions(DNSSEC)provideauthenDcaDonandensuretheintegrityoftheDNSthroughtheuseofcryptographicsignaturesgeneratedwithpublickeytechnology
• ESnetusingDNSSigner,adedicatedappliancefromSecure64Corp
• CompletedinDecember–aheadofthemandatefromtheU.S.OfficeofManagementandBudget(OMB)– Top‐level.govdomainshadtobesignedbyFebruary2009– Thoseimmediatelyunderthe.govdomainhadtoimplementDNSSEC
bytheendof2009
SpectrumUpgrade
• Spectrumr9.1:– AllJuniperroutersarebeingpolledwithandsendtrapsusingIPv6– TheSpectrumMPLStransportmanagerauto‐discoverstheOSCARS
circuittopology.
– AddiDonalthresholdingalarmsforinterfaceuDlizaDon,errorsandrouteenginetemperature.
– OSCARSLSPalarmshavenowbeenintegratedintothedailyoutagesonthePlannedMaintenanceCalendar(PMC)
• ThesemeasurementswillprovideabasisforOSCARSavailabilitymetrics
CfengineInstallaDon
• NeededtoprovideautomatedconfiguraDonandmaintenanceofservers,fromapolicyspecificaDon– Everymeasurementhost(60+)mustbemaintainedinaknowngood
statesinceOS&soVwaredifferencesaffectmeasurementresults
– SoVwaresystemsrunning(perfSONAR,OSCARS)areunderacDvedevelopment&mustbeupgradedfrequently
• DeploymentandconfiguraDonunderwayforLinuxandFreeBSDhosts– AutomatenewOSinstallaDonsonalargesetofhosts– Automatedpatching
– ConfiguraDonmanagementverificaDon&reporDng
BladeServerDeployment
• Needed a way to effectively manage growing number of servers (250+) – Increasing complexity, cost, power consumption, demands on
staff
• Phased deployment over next year • Benefits:
– Reduce rack space consumed up to 80% – Reduce power consumption up to 65% – Eliminates near-term need for HVAC and power upgrades in
ESnet datacenter (estimated at $2M+) – Automated management reduces FTE costs
Nagios&OpsView
• AnindustrystandardmonitoringsystemthatenablesorganizaDonstoidenDfyandresolveITinfrastructureproblemsbeforetheyaffectcriDcalbusinessprocesses– Neededawaytomonitorserversandotherequipment(i.e.videoMCUs)
– ImplementaDonisautomatedandfullyredundant• AutomaDcreporDngofoutages,availability,health,response,etc.
CommunityEngagement
• NetworkRequirementsWorkshops– AdvancedScienDficCompuDng
• PublishedDec2:h6p://www.es.net/hypertext/requirements.html– HighEnergyPhysics
• WorkshopCompleted:Aug28,2009• Reportundergoingfinalreview
– BER/BESscheduledfor2010• DICE(DANTE,Internet2,Canarie,ESnet)
– ConDnuetocollaborateonvirtualcircuits,perfSONAR– DICEFrameworkagreement
• Technicaldiscussionswithothernetworks– Surfnet,CERN,Nordunet,RNP,AIST,etc.
• ContribuDonstorelevantpublicstandardbodies– LeverageoperaDonalandnetworkresearchexperiences
• CurrentfocusandparDcipaDon– GLIF(GlobalLambdaIntegratedFacility)
• Fenius• AdvancedGOLEiniDaDve
– OGF(OpenGridForum)• NM:NetworkMeasurements
• NMC:NetworkMeasurementandControl
• NSI:NetworkServiceInterface• NML:NetworkMarkupLanguage• GHPN:GridHighPerformanceNetworkResearch
AdvancingStandards
StaffProjects
PMC,Twi6er,Weathermap,View,etc.
PlannedMaintenanceCalendar
• Developedin‐houseinresponsetoaDOErequirementtocategorizeplannedvs.unplannednetworkserviceoutages– Improves the quality of ESnet notifications and saves ESnet
operators significant time and effort
• UsesSpectrumNetworkManagementSystem– Providesdetailedoutagedata– MaintenanceeventschedulingandcorrelaDon
– Path‐basedserviceavailability– MaintenanceimpactpredicDon
– TargetednoDficaDon– Topologymapping
Contact:MikeO’[email protected]
PassiveNetworkMeasurement
• ESnethasreleasedtwoporDonsofit'sSNMPdatacollecDonsystemasopensourcesoVware:TSDBandESxSNMP– TSDB(TimeSeriesDataBase)isadatabaseopDmizedforstoringlarge
amountsofDmeseriesdata
– ESxSNMP(ESneteXtensibleSNMPsystem)isaflexibleSNMPpollingsystemwhichhasbeendesignedforhighreliabilityandminimalupkeep.ESxSNMPusesTSDBtostoreitsDmeseriescounterdata.
– BothareavailableonGoogleCodeContact:[email protected]
2/3/10
• ExploringhowtoprovideimmediatestatusmessagesdirectlyfromESnetalertsystemsastheyoccur
• Followersofthesetopicalcommunityfeedsreceivereal‐Dmeconciseandmeaningfulmessagesaboutserviceimpactandthestateofthenetwork.
• InthespiritofTwi6er,messageswillberelevantattheinstanttheyaresent,nottomorrowornextweek,rightnow
• PotenDalTwi6erFeeds:– Customerserviceimpact–persite– Carriercircuitoutages– OSCARSvirtualcircuitoutages– Peeringoutages
Contact:MikeO’[email protected]
Twi6erFeeds
• BycollaboraDngwithotherR&Epartners,ESnethasconstructedadisplaythatcapturesbandwidthuDlizaDonofeachcircuitandgraphicallydisplaysthisinformaDonforeaseofviewing
• NetworktrafficstaDsDcsbyinterface,network(IP&SDN),peeringconnecDon– Showsaverageandmaximumusageoverperiodfrom30secsto60
mins
– ShowsOSCARScircuittopologyandreservedbandwidth• 24hrplaybackcapability• h6p://www.weathermap.es.net/testContact:[email protected]
ESnetWeathermap
• GoogleEarth‐basednetworkvisualizaDontool– LayerednetworkinformaDon
• Fiberplant,opDcallayer,rouDng/switchlayer– LinkstoESnetdatabases
• Hadmanydifferenttoolsdevelopedovertheyears– AccessinginformaDonrequiredmanysteps,processslow– Neededbe6erinformaDonmanagementforoperaDonalefficiency
Contact:[email protected]
ESnetView
