computing infrastructure
DESCRIPTION
Computing Infrastructure. R.Dubois [email protected]. Potential Work Areas. Management of simulations Sources of cycles Manpower to run them workflow engine to minimize effort? Managing the data that results: catalogue and access Look at usability of tools - PowerPoint PPT PresentationTRANSCRIPT
DESC mtg U Penn 11-13 June, 2012
Computing Infrastructure
Computing Parallel Session
DESC mtg U Penn 11-13 June, 2012
Potential Work Areas
• Management of simulations– Sources of cycles– Manpower to run them
• workflow engine to minimize effort?– Managing the data that results: catalogue and access
• Look at usability of tools– Connection between cosmo sims and ImSim– Harnesses, config tools for running ImSim– Long term support for ImSim?
• Collaborative tools: foster communications and sharing– Forum, wiki
Computing Parallel Session
DESC mtg U Penn 11-13 June, 2012
Managing Simulations
• Clusters at collaboration sites• Grid cycles• John asking for 3 FTEs to run sims
• Data catalogue to track simulated data– Probably no off-the-shelf tool currently to track data across arbitrary
sites and provide access tools– Examples from Fermi:
• Catalogue – being adapted to LSST camera test data as demo• Workflow engine – web based submission/tracking tool
– Central submission to multiple sites with addition of simple interface to local batch system
– Being used at SLAC for ImSim runs now as demo
Computing Parallel Session
DESC mtg U Penn 11-13 June, 2012
Catalogue Example
Computing Parallel Session
DESC mtg U Penn 11-13 June, 2012
Example Work Flow Engine
Computing Parallel Session
DESC mtg U Penn 11-13 June, 2012
Running ImSim
Computing Parallel Session
User Friendliness:• 10 CPU-hrs per visit• Several steps
Understand DM tools:• Best path to catalogues
DESC mtg U Penn 11-13 June, 2012
ImSim Usability Aids
• Have a recipe for running – needs user intervention between steps– Can we streamline the recipe?
• Might the code benefit from profiling? May be some low hanging fruit to make it go faster than 10 hrs per visit?
• Might not be hard to keep a repository of compiled versions of ImSim for popular OSes– Fetch everything needed via an installer
• If we hope for a broader developer community– Maybe look at packaging and internals to enable/encourage new
developers to get involved
Computing Parallel Session
DESC mtg U Penn 11-13 June, 2012
Collaborative Tools
• Common repository for collaboration code– Can create a mix of “user contributions” which can evolve into a
collection of integrated tools in the same repository
• Ease of collaboration:– Use a forum for discussion
• More powerful than mailing lists
• Use good wiki tool for sharing and discussing documents, analyses etc
Computing Parallel Session
DESC mtg U Penn 11-13 June, 2012
Some Open Questions
• Can there be a common framework defined for analysis tools? At what level? FITS atomic model with files as interfaces? What does LSST project framework look like? Can/should it be emulated?
• Should there be common tools defined for things like plotting etc?• Any connection to other surveys? How to share/store other surveys'
data?• What about manpower for supporting all this?• Analysis tasks & systematics session• Verification & Validation of project pipelines• Plea for cross experiment computing resources• What is 'deliverable' for systematics studies? Could there be data flow
diagrams to help identify any?• Non-ImSim sims??
– Need fast sim capability
Computing Parallel Session