hep@home a distributed computing system based on boinc september - chep 2004 pedro andrade antónio...
TRANSCRIPT
![Page 1: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/1.jpg)
HEP@HomeA Distributed Computing System Based on BOINC
September - CHEP 2004
Pedro Andrade <[email protected]>
António Amorim <[email protected]>
Jaime Villate <[email protected]>
![Page 2: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/2.jpg)
October 30th, 2004 HEP@Home 2
Overview
• Introduction
• BOINC
• HEP@Home
• ATLAS Use Case
• Tests and Results
• Conclusions
![Page 3: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/3.jpg)
October 30th, 2004 HEP@Home 3
Introduction
• Project participants:– Faculdade de Ciências da Universidade de Lisboa– Faculdade de Engenharia da Universidade do Porto
• From Grid-Brick system presented at CHEP2003• Goals:
– Create a distributed computing system– Explore commodity CPU’s and disks and keep them
together– Use public computing– Evaluate its use for dedicated HEP clusters.
![Page 4: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/4.jpg)
October 30th, 2004 HEP@Home 4
Overview
• Introduction• BOINC
– Description– Features– Behavior– Related Work
• HEP@Home• ATLAS Use Case• Tests and Results• Conclusions
![Page 5: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/5.jpg)
October 30th, 2004 HEP@Home 5
Description
• Stands for Berkeley Open Infrastructure for Network Computing
• Generic software platform for distributed computing
• Developed by the SETI@Home team• Based on public computing• Key concepts
– Project– Application– Workunit (Job)– Result
![Page 6: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/6.jpg)
October 30th, 2004 HEP@Home 6
Features
• Generic platform: supports many applications / projects
• Projects can be run simultaneously
• Common language applications can run as BOINC applications
• Fault-tolerance
• Monitored through a Web interface
• Implements security mechanisms
![Page 7: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/7.jpg)
October 30th, 2004 HEP@Home 7
Behavior
• Initial communication• Work request
– Hardware characteristics
• Server decides• Workunit download
– Application– Input files
• Results Upload
Client makes requests, Server is passive
![Page 8: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/8.jpg)
October 30th, 2004 HEP@Home 8
Related Work
• Project-specific solutions:– SETI@Home– Distributed.net– Folding@Home
• Commercial solutions
• XtremWeb
• JXGrid
![Page 9: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/9.jpg)
October 30th, 2004 HEP@Home 9
Overview
• Introduction• BOINC• HEP@Home
– Background– Additional Features– Behavior
• ATLAS Use Case• Tests and Results• Conclusions
![Page 10: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/10.jpg)
October 30th, 2004 HEP@Home 10
Background
Grid-Brick project:• Presented at CHEP2003• Goal was merge storage
units with computing farms.
• Conclusions:– No central resource
manager– Plug and play clients– Increase robustness– Fault-tolerant system
![Page 11: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/11.jpg)
October 30th, 2004 HEP@Home 11
Additional Features
• Avoid data movement
• User specific applications
• Environments– Scripts– Libraries
• Environments patches
• “get input” apps• Job dependencies
![Page 12: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/12.jpg)
October 30th, 2004 HEP@Home 12
Behavior
• Initial communication• Work request
– Hardware characteristics– Available input files
• Server decides:– Input file exists: ok– No input file: wait, run "get
input" app
• Workunit download:– Application– Environment / Patches
• Results Upload
![Page 13: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/13.jpg)
October 30th, 2004 HEP@Home 13
Overview
• Introduction
• BOINC
• HEP@Home
• ATLAS Use Case
• Tests and Results
• Conclusions
![Page 14: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/14.jpg)
October 30th, 2004 HEP@Home 14
ATLAS Use Case
• How can physicists use HEP@Home to run ATLAS jobs.
• The actors of this use case can be:– Physicist doing personal job submission– Real production
• Let us suppose we have:– Several ATLAS jobs to run– We know what files each job will produce and
consume and how to generate or get these files. – We have computers connected to the Internet
![Page 15: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/15.jpg)
October 30th, 2004 HEP@Home 15
ATLAS Use Case
• Execution Steps:– Select or submit ATLAS application– Work submission:
• environment files (job options files, scripts, etc)• environment patch• input file template• "get input" application• result (output file) template
• As a result the user gets the aggregation of the produced output files as a unique output file.
![Page 16: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/16.jpg)
October 30th, 2004 HEP@Home 16
Overview
• Introduction
• BOINC
• HEP@Home
• ATLAS Use Case
• Tests and Results
• Conclusions
![Page 17: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/17.jpg)
October 30th, 2004 HEP@Home 17
Tests
• Based on the defined ATLAS Use Case• Typical ATLAS jobs sequence using Muon events:
– Generation: e events (1x)– Simulation: e/10 events (10x)– Digitization: e/10 events (10x)– Reconstruction: e/10 events (10x)
• Two groups of tests were defined: e = 100, e = 1000.• For each group, 4 tests were made:
– One simple client– Two BOINC client– Four BOINC client– Eight BOINC client
![Page 18: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/18.jpg)
October 30th, 2004 HEP@Home 18
Results - Execution Times
• Group A: 100 events• Group B: 1000 events
![Page 19: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/19.jpg)
October 30th, 2004 HEP@Home 19
Results - Data Movement
• 1000 events in 8 machines:• Seqx: events x00-x99
![Page 20: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/20.jpg)
October 30th, 2004 HEP@Home 20
Overview
• Introduction
• BOINC
• HEP@Home
• ATLAS Use Case
• Tests and Results
• Conclusions
![Page 21: HEP@Home A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate](https://reader035.vdocuments.net/reader035/viewer/2022062718/56649e6c5503460f94b6b385/html5/thumbnails/21.jpg)
October 30th, 2004 HEP@Home 21
Conclusions
• Several BOINC projects are currently running successfully worldwide
• From HEP@Home tests:– Execution of user applications => more flexibility– Environments and patches => easier work submission– Heavier computation => better results– Low data movement => better results
• HEP@Home can be brought to physicists daily tasks with not much effort