qcdgrid ukqcd achievements and future priorities who and what achievements qcdgrid middleware future...
Post on 28-Dec-2015
212 Views
Preview:
TRANSCRIPT
QCDgrid
UKQCD Achievements and Future Priorities
•Who and what
•Achievements
•QCDgrid middleware
•Future priorities
•Demo of meta-data catalogue browser
Alan IrvingUniversity of Liverpool
UKQCD
January 2003
QCDgridUKQCD and the Grid: QCDgrid architecture
•Phase 1: data grid
•Phase 2: distributed processing
•Roll out to all UKQCD
•ILDG: International Lattice Data Grid
(ILDG meeting in Edinburgh: Dec 19/20)
January 2003
QCDgridQCDgrid people
The main players……
James Perry – EPCCjamesp@epcc.ed.ac.uk
– Globus, EDG and QCDgrid middleware, browser (OGSA)
Chris Maynard – Edinburghcmaynard@ph.ed.ac.uk
– XML schema, QCDgrid administrator
Craig McNeile – Liverpoolmcneile@amtp.liv.ac.uk
– UKQCD Software Manager, QCDgrid user interface
January 2003
QCDgridQCDgrid Achievements
• We have a working data grid (J Perry, EPCC)• We have used it for routine work (C McNeile,
Liverpool)• We can build our own 0.6 Tbyte RAID disk
arrays for < £2000 (S Downing, Liverpool)• We have explicitly tested RAID systems by
reformatting one drive (C Allton , Swansea)• We have a draft XML schema for lattice QCD
data (C Maynard, Edinburgh)• We have started an International Lattice Data
Grid project (R Kenway, Edinburgh)
January 2003
QCDgridQCDgrid middleware (Globus 2.0, Linux 7.x)
User commands•qcdgrid-list
•put-file-on-qcdgrid
•get-file-from-qcdgrid
•i-like-this-file
•qcdgrid-delete
Configuration•qcdgrid.conf
•nodes.conf
•nodeprefs.conf
Grid administration•control-thread.sh
•add-qcdgrid-node
•disable-qcdgrid-node
•enable-qcdgrid-node
•remove-qcdgrid-node
•retire-qcdgrid-node
•unretire-qcdgrid-node
•get-disk-space
•qcdgrid-checksum
•qcdgrid-filetime
•rebuild-qcdgrid-rc
•create-qcdgrid-rc
•delete-qcdgrid-rc
•verify-qcdgrid-rc
January 2003
QCDgrid
QCDgrid: the future
January 2003
QCDgrid QCDOC and the grid …
UKQCD is a distributed physics organisation
• Distributed computing Grid
• Local disk farms and clusters
• QCDOC : source of configs
Primary data from QCDOC
• QCDOC not directly on the grid
• Output -> binary + XML
• Front end feeds data to QCDgrid
Data storage on RAID
• RAIDs as QCDgrid nodes
Analyses on clusters
• Phase I: analysis on specified nodes pulling/pushing data from QCDgrid
• Phase II: cluster processing via grid middleware?Node
QCDOC
FE
Grid
January 2003
QCDgridQCDgrid: future plans
Short term
• Decide on database: native XML (Xindice) or relational + XML interface tools
• Complete meta-data catalogue browser
• Expand no of sites in Grid to 4 UKQCD + RAL
• Decide on data-binding or other strategy for QCDOC I/O
• Construct more RAID arrays.
Longer term
• Agree ILDG standard XML schema
• Agree ILDG middleware functionality at web-services level
• Monitor EDG M/W• Investigate EDG M/W for
remote job processing • Test interoperability with
other ILDG sites• Develop M/W to implement
any new IDLG strategy
January 2003
QCDgrid
QCDgrid and HPC resources requested
Hardware• ‘Tier 1’: QCDOC + FE [JIF]• ‘Tier 2’: [PPARC, ‘son-of-SRIF’]
– 6x100 node PC clusters – 6x20 Tbyte disk farms
(could be integrated with LHC Tier 2 facilities?)
Staff• Grid specific: [PPARC/GridPP, Esci]
– 2 FTEx3years for QCDgrid and ILDG development– 1 FTEx3 years for maintenance, dissemination (shared with other theory groups)
• Core SW development: [PPARC/HPC]– 4 phys. progs. across all UKQCD
• Subdetector specific:– N.A.
January 2003
QCDgridQCDgrid logical name browser
January 2003
QCDgridQCDgrid metadata catalogue browser
January 2003
QCDgridOutput from Metadata Catalogue search
trumpton.ph.ed.ac.uk:aci|aci> Query returned 427 results
Grid filename: NF2/BETA526/CLOVER195/V16X32/KAPPA3450/GAUGE/D526C195K3450U017500.tar
Grid filename: NF2/BETA526/CLOVER195/V16X32/KAPPA3450/GAUGE/D526C195K3450U017600.tar
Grid filename: NF2/BETA526/CLOVER195/V16X32/KAPPA3450/GAUGE/D526C195K3450U017700.tar
Grid filename: NF2/BETA526/CLOVER195/V16X32/KAPPA3450/GAUGE/D526C195K3450U017800.tar
Grid filename:
NF2/BETA526/CLOVER195...
January 2003
QCDgridQCDgrid command line use (get-file-from-qcdgrid)
ulgbcm.liv.ac.uk:aci|gridwork> getmoreconfigscreating an initial todo list7 configs in initial listtodo list shows 7 configs still to dobut 5 have .tar already done, so delete any duplicatesupdating todolistnow 2 configs still to be donegettingNF2/BETA52/CLOVER202/V16X32/KAPPA3500/GAUGE/D52C202K3500U014060.tar
from the gridgettingNF2/BETA52/CLOVER202/V16X32/KAPPA3500/GAUGE/D52C202K3500U014070.tar
from the gridnow have:--rw-r--r-- 1 aci aci 42139648 Dec 11 21:41
/users/aci/configs/D52C202K3500U014010.tar-rw-r--r-- 1 aci aci 42139648 Jan 27 21:07
/users/aci/configs/D52C202K3500U014020.tar-rw-r--r-- 1 aci aci 42139648 Jan 27 21:52
/users/aci/configs/D52C202K3500U014030.tar
top related