transparently gathering provenance with provenance aware condor christine reilly and jeffrey...
TRANSCRIPT
Transparently GatheringProvenance
with Provenance Aware Condor
Christine Reilly and Jeffrey NaughtonDepartment of Computer Sciences
University of Wisconsin - Madison
TaPP ‘09 February 23, 2009
2
Motivations for Our Work
• Scientific Computing
• Grid Computing
• Condor job scheduling system
3
Motivation: Scientific Computing
• What input data was used to produce this output data?
• Example:– Blast sequence DB is updated periodically.– Did my last computation use the latest
version of the Blast DB?
4
Motivation: Scientific Computing
• Need to find the root of anomalous results.
• Example:– Multiple members of a research group can
update the code for the simulation.– Did two researchers use the same version
of the executable?
5
Motivation: Grid Computing
• What grid resources did my computation use?
• Example:– A machine in the grid has a hardware
problem (bad DIMM, corrupt disk).– Did any of my computations use the bad
machine?
6
Motivation: Condor
• Condor:– Distributed computing system.– Runs jobs from wide range of applications
and fields of study.
• Quill: Captures operational information exposed by Condor.
• Could Quill be used for provenance?– Users would get provenance “for free”.
7
Provenance System Goals
• Generic: Can be used by many different applications.
• Transparent: Users don’t need to alter their applications.
• Can we use Condor as the execution system for this type of provenance system?
• Can we do this with minimal impact on Condor developers?
8
Our Model of Provenance
• File level granularity.• The provenance of a file is:
– All files involved in its creation– Job execution environment.
input exe output
ExecutionEnvironment
9
Outline of Talk
• Motivation and Introduction
• PAC - Provenance Aware Condor
• Is PAC Practical? (Storage requirements and overhead)
• DBLife: Benefits and Limitations of PAC
• Conclusions
10
Provenance Aware Condor
• Three parts: Condor, Quill, FileTrace• Condor is the job execution system.• Quill gathers operational information
from Condor.• FileTrace gathers information about files
used by Condor jobs.• Provenance is obtained by querying the
Quill database.
11
What is Condor?
• Provides a simple interface to a cluster of machines.– User gives computing jobs to Condor.– Condor finds available machine that meets job
requirements.– Condor ensures that the job completes execution,
even in the presence of failures.
• Used for many different applications:– Physics, Biology, Chemistry, Computer Science,
Engineering applications.– Academic research, national labs, industry.
12
What is Condor? (part 2)
• Research project at UW, but also resembles commercial software.– 35 faculty, full-time staff, students.– Used at ~2000 locations world-wide.– Running on more than 250,000 machines.
• Annual user conference attracts hundreds of attendees.
13
SubmitMachine
ExecuteMachine
CentralManager
User’sMachine
User’sLocalFiles
SharedFile
System
JobSpecs Exe, In
Files
OutputFiles
Output Files
Executable, Input Files
MatchInfo
MatchInfo
JobRequirements
MachineResources
Condor Overview
14
What is Quill?
• Gathers operational data from Condor.– Improves performance of user’s inquiries
about their jobs.– Information about Condor operations is
stored in RDBMS.
• Created by group of database researchers (including us).
• Ships with Condor.
15
Quill’s Design Requirements
• Expose Condor operational data for querying through a DBMS interface.
• Cannot affect how Condor operates:– No changes to the existing Condor code.– No change to how Condor is used.
• Failures in Quill cannot cause Condor to fail.
16
SubmitMachine
ExecuteMachine
CentralManager
User’sMachine
User’sLocalFiles
SharedFile
System
JobSpecs Exe, In
Files
OutputFiles
Output Files
Executable, Input Files
JobRequirements Match
InfoMatchInfo
MachineResources
Quill Overview
17
File Access in Condor
• Three methods of accessing files:– Remote system calls– Condor File Transfer– Shared file system
• Quill’s information about files:– Detailed information about File Transfers– Limited information about some other files– No guarantee that it detects all files
• Problem: Because Condor does not track all file access, Quill may miss file information.
• FileTrace gathers information about all files.
18
FileTrace
• Modification of UNIX strace.• Transparent to Condor.• For each file open or close system call
FileTrace gathers:– Condor Job Id.– File information: pathname, last modified time,
checksum, size.– File access information: activity type, open flags,
file pointer.
19
Provenance in PAC
• Provenance of a file is:– All files involved in its creation: Executable,
Input files, Libraries and system files.– Job execution environment: when, on what
machine.
• PAC has no control over files in the user’s file system.
20
Provenance Queries in PAC
What files were used by this job?
SELECT pathname, activity_type, open_flags, last_modified, size, checksum
FROM filetraceWHERE globaljobid=‘my_job_id’;
pathname activity type open flags last
modifiedsize
(bytes) checksum
program.exe exe 2008-10-06 17:52:16-05
687 8FC8316B8930835C9085E 953F2D36745C2877346
/path/to/input.xml open O_RDONLY 2008-10-06 17:52:28-05
9554599 57B7A53ACF390B1928A6 C4B6E935DD9F2CBE9F46
/path/to/output.xml close 2008-10-06 17:55:19-05
2E+07 9D32885FDCD9171E44F9 302BB0141DFC1FC6B772
21
Provenance Queries in PAC
More questions PAC can answer:
• Were any of my files created on the bad machine?
• Was this output created using the current versions of the input and executable files?
• Did any input files change between two runs of the application?
22
Outline of Talk
• Motivation and Introduction
• PAC - Provenance Aware Condor
• Is PAC Practical? (Storage requirements and overhead)
• DBLife: Benefits and Limitations of PAC
• Conclusions
23
Is PAC Practical?
• Users will not tolerate PAC if:– Data storage requirements are too
onerous.– Computational overhead is significant.
24
Estimate Table Growth
• Use the run time and number of files used by seven scientific applications (BLAST, IBIS, CMS, Nautilus, Messkit Hartree-Fock, AMANDA, SETI@home).
• Simulate running each application constantly for a year on a 1000 machine cluster.
• How big is the FileTrace table?
25
Size of File Trace TableApplication Continuously Run for 1 year on 1000 Machines
0
0.5
1
1.5
2
2.5
3
3.5
IBIS SETI CMS Nautilus Amanda HF Blast
Log Table Size (GB)
1.15 TB900 GB
9 GB
30 GB41 GB
190 GB
540 GB
26
Test Cluster for Overhead
• Machine Specs:– Pentium 2.40 GHz Core 2 Duo– 4GB RAM– Two 250GB SATA-I hard disks.
• Condor Cluster: – Central Manager– Submit Machine– Database Machine– 10 execute machines
27
Overhead of PAC
• Use synthetic program.– Writes random numbers to files.– Can vary the number of files generated
and run time.
• 4 sets of jobs with run time and number of files based on the scientific programs from the size experiment.– Run time per job: 5 or 20 minutes.– Files per job: 10 or 300.
28
Overhead of PAC
• Two Clusters: PAC and Condor without Quill or FileTrace.
• For each set of jobs, submit as many jobs as will take ~12 hours to complete.
• No significant difference in average time per job or total run time.
29
Outline of Talk
• Motivation and Introduction
• PAC - Provenance Aware Condor
• Is PAC Practical? (Storage requirements and overhead)
• DBLife: Benefits and Limitations of PAC
• Conclusions
30
DBLife
• Community Information Management (CIM)
• Creates and maintains an ER graph of the database research community.
• Gathers data by crawling web.
• Interesting example for PAC:– Large, complex workflow.– Accesses many files.
31
Overview ofDBLife
List of Sources
DomainKnowledge
Provided byCommunity
Expert
Data Page Layer
Mention Layer
Conference
University
PaperCompany
JoeER Layer
Joe
Jane
DBLifeHome
CIMApplication
32
DBLife Front Page
33
DBLife Person Homepage
34
Questions PAC Can Answer
• In general - questions developers and system administrators would ask.
• Examples:– Is this the version of the program that was
used to create the current data set?– Was this machine used to create any part
of the DBLife data set?– Does the DBLife ER model reflect the
recent changes on my web page?
35
Questions PAC Cannot Answer
• In general - questions that require knowledge of the semantics of DBLife.
• Examples:– Why does the DBLife portal show that
person X is affiliated with institution Y?– Why doesn’t my page on the DBLife portal
include that I presented a paper at this workshop?
36
Problem: Large FileTrace Table
• Ran DBLife with 10 crawl start points:– FileTrace table: 100 MB
• Full scale run uses 1000 start points– FileTrace table est.: 10 GB
• Run DBLife daily for 1 year:– FileTrace table est.: 3.7 TB
• Possibilities for reducing table size:– Could be smarter about how data is stored.– Table compression.
37
Outline of Talk
• Motivation and Introduction
• PAC - Provenance Aware Condor
• Is PAC Practical? (Storage requirements and overhead)
• DBLife: Benefits and Limitations of PAC
• Conclusions
38
Limitations of PAC
• File level is coarse granularity.
• Files are not under PAC’s management.
• Workflows are not explicitly recorded.
• Doesn’t know about the semantics of an application.
39
Contributions of PAC
• Many different applications can use PAC for provenance.
• Has little impact on Condor.
• Could be combined with other systems to yield a stronger provenance system.– Workflow management system.– File management system.
40
Future Work
• Integrate PAC with an Xlog version of DBLife.– Xlog is similar to Datalog– Record information when Xlog rule is
triggered.– Provides fine grained provenance.– Can answer questions that depend on the
semantics of the application.
41
Acknowledgements
• Supported in part by National Science Foundation Award SCI-0515491.
• Thank you CondorDB, DBLife, and Condor groups!