network monitoring with r-gma in edg paul mealor researcher
DESCRIPTION
Network monitoring with R-GMA in EDG Paul Mealor Researcher. JRA4 R-GMA meeting, 22 July 2004. www.eu-egee.org. EGEE is a project funded by the European Union under contract IST-2003-508833. Contents. Disclaimer Monitoring tools and modifications Probes Coordination Protocol - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Network monitoring with R-GMA in EDG Paul Mealor Researcher](https://reader035.vdocuments.net/reader035/viewer/2022062221/56814871550346895db57c03/html5/thumbnails/1.jpg)
EGEE is a project funded by the European Union under contract IST-2003-508833
Network monitoring with R-GMA in EDG
Paul Mealor
Researcher
JRA4 R-GMA meeting, 22 July 2004
www.eu-egee.org
![Page 2: Network monitoring with R-GMA in EDG Paul Mealor Researcher](https://reader035.vdocuments.net/reader035/viewer/2022062221/56814871550346895db57c03/html5/thumbnails/2.jpg)
JRA4 R-GMA meeting, 22 July 2004 - 2
Contents
• Disclaimer
• Monitoring tools and modifications
• Probes Coordination Protocol
• Network Cost Estimation System
• Schemas
• Related work
![Page 3: Network monitoring with R-GMA in EDG Paul Mealor Researcher](https://reader035.vdocuments.net/reader035/viewer/2022062221/56814871550346895db57c03/html5/thumbnails/3.jpg)
JRA4 R-GMA meeting, 22 July 2004 - 3
Disclaimer
• R-GMA components have changed a lot since EDG Streaming producer Primary producer (memory) Consumer Consumer Archive, database Well, it’s changed
![Page 4: Network monitoring with R-GMA in EDG Paul Mealor Researcher](https://reader035.vdocuments.net/reader035/viewer/2022062221/56814871550346895db57c03/html5/thumbnails/4.jpg)
JRA4 R-GMA meeting, 22 July 2004 - 4
Measurementwrapper
script
Monitoring tools and modifications
• PingER (which begat IperfER and UDPMon)
• Wrapper script run periodically By Cron, and latterly PCP
• Runs ping, captures output
• Output stored in flat text files
• Web user interface Displays graphs and tables Also, unformatted data available
Web UI
Text files
Ping
![Page 5: Network monitoring with R-GMA in EDG Paul Mealor Researcher](https://reader035.vdocuments.net/reader035/viewer/2022062221/56814871550346895db57c03/html5/thumbnails/5.jpg)
JRA4 R-GMA meeting, 22 July 2004 - 5
Measurementwrapper
script
Monitoring tools and modifications
Web UI
Text files
Ping netmon2rgmad
StreamProducer
Servletson
dedicatedmachine
MySQLArchive
Archiver
DBProducer
Consumer
netmonarchiverd
![Page 6: Network monitoring with R-GMA in EDG Paul Mealor Researcher](https://reader035.vdocuments.net/reader035/viewer/2022062221/56814871550346895db57c03/html5/thumbnails/6.jpg)
JRA4 R-GMA meeting, 22 July 2004 - 6
GridFTP daemon(gsiwuftpd)
Monitoring tools and modifications
• GridFTP daemon
• Can be set to write logs
• ftlog2rgmad reads logs for updates File gets longer: new entries at end File gets shorter: everything’s a new entry
• Publishes new entries via Stream Producer
Log
ftlog2rgmad
StreamProducer
Servletson
dedicatedmachine
![Page 7: Network monitoring with R-GMA in EDG Paul Mealor Researcher](https://reader035.vdocuments.net/reader035/viewer/2022062221/56814871550346895db57c03/html5/thumbnails/7.jpg)
JRA4 R-GMA meeting, 22 July 2004 - 7
Probes Coordination Protocol
• Full-mesh measurements impossible
• Hand-built, centralised schedules don’t work
• PCP automates scheduling
• Token passed between nodes in a clique
name:iperfmember:ccwp7.in2p3.frmember:adc0003.cern.chmember:grid001f.cnaf.infn.itmember:gppnm.gridpp.rl.ac.ukmember:mon001.fzk.deperiod:1800timeout:600delay:60option:ExternalLockcommand:edg-iperf iperf
• Nodes with the token can make measurements
• Many cliques can coexist Allows hierarchies to be set up ExternalLock option: no other commands while this one is running
edg-pcp-extern-lock program to lock a pair of hosts
![Page 8: Network monitoring with R-GMA in EDG Paul Mealor Researcher](https://reader035.vdocuments.net/reader035/viewer/2022062221/56814871550346895db57c03/html5/thumbnails/8.jpg)
JRA4 R-GMA meeting, 22 July 2004 - 8
Probes Coordination Protocol (2)
• Fault tolerance Token should traverse the clique in period seconds If it left a node period + timeout seconds previously a new token is
generated• Protects against lost tokens
Duplicate tokens are dropped
• Configuration updates Are easy: the token contains the
configuration
name:iperfmember:ccwp7.in2p3.frmember:adc0003.cern.chmember:grid001f.cnaf.infn.itmember:gppnm.gridpp.rl.ac.ukmember:mon001.fzk.deperiod:1800timeout:600delay:60option:ExternalLockcommand:edg-iperf iperf
![Page 9: Network monitoring with R-GMA in EDG Paul Mealor Researcher](https://reader035.vdocuments.net/reader035/viewer/2022062221/56814871550346895db57c03/html5/thumbnails/9.jpg)
JRA4 R-GMA meeting, 22 July 2004 - 9
Network Cost Estimation System
• Implemented for C++ and Java
• Multiple backends MySQL R-GMA (ish :-/ ) LDAP
• Extra information: Mapping tables: NetworkCE and NetworkSE Match all SE/CEs to a Network Monitor
NetworkCost getNetworkCost(source,dest,filesize)
NetworkCost[][] getNetworkCosts(sources[],dests[],filesize)
netmon-rgma-info
StreamProducer
![Page 10: Network monitoring with R-GMA in EDG Paul Mealor Researcher](https://reader035.vdocuments.net/reader035/viewer/2022062221/56814871550346895db57c03/html5/thumbnails/10.jpg)
JRA4 R-GMA meeting, 22 July 2004 - 10
Network Cost Estimation System
• To be done: Make use of the GridFTP data Actually use R-GMA to query the Archive
![Page 11: Network monitoring with R-GMA in EDG Paul Mealor Researcher](https://reader035.vdocuments.net/reader035/viewer/2022062221/56814871550346895db57c03/html5/thumbnails/11.jpg)
JRA4 R-GMA meeting, 22 July 2004 - 11
Schemas
NetworkTCPThroughputNMIdSourceNMIdDestinationtoolbufferSizestreamsdurationtimevalue
NetworkCE
CEIdNMId
NetworkRTT
NMIdSourceNMIdDestinationtoolpacketSizetimeminimummaximumaverage
NetworkCE
CEIdNMId
NetworkSE
SEIdNMId
ComputingElement
CEIdGRAMVersionArchitectureOpSys&c…
1
1
1
1
![Page 12: Network monitoring with R-GMA in EDG Paul Mealor Researcher](https://reader035.vdocuments.net/reader035/viewer/2022062221/56814871550346895db57c03/html5/thumbnails/12.jpg)
JRA4 R-GMA meeting, 22 July 2004 - 12
Related work
• Glue Schema Almost identical schemas (as they were based on ours) Measurements associated with a “Network Element” Network Element is a path between two nodes, with a particular
QoS and so on
• NMWG How do these XML schemas translate?
• In WP7 MapCenter (visualisation of host status) TopoGrid (visualisation of paths)
![Page 13: Network monitoring with R-GMA in EDG Paul Mealor Researcher](https://reader035.vdocuments.net/reader035/viewer/2022062221/56814871550346895db57c03/html5/thumbnails/13.jpg)
JRA4 R-GMA meeting, 22 July 2004 - 13
END
• http://www.hep.ucl.ac.uk/~pdm/edg/docs/
• EDG WP7: http://ccwp7.in2p3.fr/
• PCP/GNMA: see WP7 final deliverables