![Page 1: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/1.jpg)
21 Sep 2005 LCG's R-GMA Applications
R-GMA and LCG
Steve Fisher & Antony Wilson
![Page 2: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/2.jpg)
![Page 3: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/3.jpg)
21 Sep 2005 LCG's R-GMA Applications
A unified approach to I & M
• Provides information on both:– The Grid itself
• Mainly for the middleware packages, but the user may also query it to understand the status of the Grid
– Grid applications• For users
• Flexible infrastructure – All data has a timestamp enabling its use for
monitoring– Dynamic addition and deletion of information
producers– Dynamic creation and deletion of tables in schema– Able to cope with failures in a distributed environment
![Page 4: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/4.jpg)
21 Sep 2005 LCG's R-GMA Applications
R-GMA – one cloud
• A relational implementation of GMA (from GGF)– Powerful data model and
query language• All data modelled as tables• SQL can express complex
queries in one expression
• Creates impression that you have one RDBMS per VO
Producer
Consumer
R-GMA
![Page 5: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/5.jpg)
ApplicationsApplications
![Page 6: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/6.jpg)
21 Sep 2005 LCG's R-GMA Applications
Job Monitoring• Requirements
– Allow grid users to monitor the progress of their jobs– Allow VO administrators to get an overview of what is
happening on the grid
• The Problem– The location where a grid job will run is not known– Worker nodes are behind firewall so are not accessible
• The Solution– R-GMA provides transport across firewalls
• The wrapper approach• Instrumenting the code
– Using a logging service– Calling R-GMA directly
![Page 7: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/7.jpg)
21 Sep 2005 LCG's R-GMA Applications
![Page 8: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/8.jpg)
21 Sep 2005 LCG's R-GMA Applications
LCG Job Monitoring
Deployed by LCG
No changes needed to user code
R-GMA Primary Producer
SP R-GMA Secondary Producer
PP
Job wrapper
PPUser code
Job wrapper
PPUser code
SPDatabase
SP
![Page 9: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/9.jpg)
21 Sep 2005 LCG's R-GMA Applications
LCG Job Monitoring
R-GMA Primary Producer
SP R-GMA Secondary Producer
PP
Deployed by LCG
Job wrapper
PPUser code
Job wrapper
PPUser code
SPDatabase
SP
L&B Server
PP
![Page 10: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/10.jpg)
21 Sep 2005 LCG's R-GMA Applications
Producers
PP
PP
PP
SP
PP - Primary ProducersInitial source of dataData published by user code stored by Producer Service
SP- Secondary ProducersUsed to republish data in order to:
co-locate information to speed up queries reduce network traffic use different producer properties
![Page 11: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/11.jpg)
21 Sep 2005 LCG's R-GMA Applications
Producer Properties
• Producers may use:– Memory
• Gives best performance for continuous queries– Database
• Best performance for joins
• Primary and Secondary Producers must support – Continuous queries
• as soon as new data becomes available it is broadcast to all interested parties
• Primary and Secondary Producers may support– History Queries
• over time sequenced data– Latest Queries
• correspond to intuitive idea of current information
![Page 12: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/12.jpg)
SEE DEMOSEE DEMO
Job Monitoring Web Interface
![Page 13: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/13.jpg)
21 Sep 2005 LCG's R-GMA Applications
Application Monitoring
Users can define their own tables
C, C++, Java and Python apis available log4j appender available
PPUser code
PPUser code
R-GMA Primary Producer
SP R-GMA Secondary Producer
PP
SPDatabase
SP
![Page 14: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/14.jpg)
21 Sep 2005 LCG's R-GMA Applications
Network Monitoring
Deployed by EGEE
Data obtained from network monitoring sensors are published via R-GMA
R-GMA Primary Producer
SP R-GMA Secondary Producer
PP
PP
Network sensor
PP
Network sensor
SPDatabase
SP
![Page 15: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/15.jpg)
21 Sep 2005 LCG's R-GMA Applications
![Page 16: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/16.jpg)
21 Sep 2005 LCG's R-GMA Applications
GridFTP Monitoring
Deployed by LCG
Data are extracted from GridFTP logs and published via R-GMA
R-GMA Primary Producer
C R-GMA Consumer
PP
C Oracle
RDBMS
PP
GridFTP log monitor
PP
GridFTP log monitor
![Page 17: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/17.jpg)
21 Sep 2005 LCG's R-GMA Applications
Consumer Service
• Query types:– History Queries
• return time sequenced data
– Latest Queries• correspond to intuitive idea of current
information
– Continuous queries• as soon as new data becomes available it is
broadcast to all interested parties
![Page 18: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/18.jpg)
SEE DEMOSEE DEMO
Grid ViewGridFTP Monitor
![Page 19: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/19.jpg)
21 Sep 2005 LCG's R-GMA Applications
Network Monitoring
Deployed by Grid Ireland
TCP logs are queried in response to queries and the results are returned via R-GMA
R-GMA Consumer
ODP R-GMA On-Demand Producer
C
CUser code
CUser code
SPTCP Log
ODP
![Page 20: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/20.jpg)
21 Sep 2005 LCG's R-GMA Applications
On Demand Producers
• Provides a Producer interface to non relational data sources
• User has to write code to handle SQL• Has been used to provide access to
TCPdump data
![Page 21: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/21.jpg)
21 Sep 2005 LCG's R-GMA Applications
R-GMA plans
• Will be taken into EGEE-II– Change from Servlets to Web Services– Add multi-VDB support– Support more than MySQL– Add authorization
• Conform to GGF/INFOD specification
![Page 22: 21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson](https://reader035.vdocuments.net/reader035/viewer/2022062511/5515c242550346a3758b46e8/html5/thumbnails/22.jpg)
21 Sep 2005 LCG's R-GMA Applications
Further Information
• R-GMA web site– http://www.r-gma.org/
• JRA1-UK web site– http://hepunx.rl.ac.uk/egee/jra1-uk/
• E-mail– [email protected]