cluster 2004 san diego, ca a client-centric grid knowledgebase george kola, tevfik kosar and miron...
TRANSCRIPT
-
Cluster 2004San Diego, CA
A Client-centric Grid Knowledgebase
George Kola, Tevfik Kosar and Miron LivnyUniversity of Wisconsin-Madison
September 23rd, 2004
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Grid TriviaHow many of you have submitted a job to the Grid resources and did never hear back from it?How many of you got mad by the inconsistent behavior of some grid resources? Completing successfully some jobs and failing others..Similar jobs performing completely different..
... We did!
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Goal: Prevent Unexpected Behavior in a GridLearn from experience and prevent them from repeating in the future again.Causes for unexpected behavior in a Grid:Black holesResources withFaulty hardwareBuggy or misconfigured softwareExtremely slow computational sitesMemory leaks ..etc
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Black holes
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Black holesDefinition: A black hole is a region of spacetime from which nothing can escape, even light.If you send a light beam to a black hole, you never hear back from it.You can only know it after you have encounter it. Is it too late?No. You should learn from experience..
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Black holes in the GridResources that accept jobs but never complete themYou send a job to a resource, but never hear back from it.
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Black hole examples from real life:In the WCER educational video processing pipeline:A specific pool was accepting and processing our jobs for a couple of hours, but evicting before completion.A machine accepted a job, but due to a memory leak it kept throwing shadow exceptions and retrying the job forever.Some thirdparty (GridFTP, DiskRouter) transfers hang occasionally and never returned. A machine caused an error because of a corrupted FPU. It successfully completed MPEG-1 encoding but failed MPEG-4.
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Grid is good.. but not perfect..Heterogeneous resourcesMulti administrative domainsSpanning wide area networks Consists of commodity hardware and softwareProne to network-, hardware-, software-, middleware- failures!
We cannot expect everything from the Grid or Grid middleware!
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Take the Ethernet ApproachA truly distributed (and very effective) access control protocol to a shared serviceClient responsible access controlClient responsible for error detectionClient responsible for fairnessKeep track of job/resource performance & failure characteristics as observed by the client.Use job/user log files collected at the client side to build a grid knowledgebase.
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Grid KnowledgebaseParse user/job log filesLoad them into a databaseAggregate experience of different jobsInterpret themPlan actionGenerate feedback to the scheduler as well as to the user
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
JOB LOGSGRID RESOURCESPersonal ComputersStorage ServersClusters
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
JOB LOGSGRID RESOURCESPersonal ComputersStorage ServersClustersGRID KNOWLEDGEBASE
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Database SchemaEvictedSubmitTerminatedAbnormallyTerminatedNormallySchedule
Execute
Job SucceededJob FailedYesNoSuspendUn-suspendUser
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Difference from existing approachesClient viewUse only job/user log files at the client sideMany administrators do not want to share resource/scheduler log files.We do not need to know everything going on in the whole gridScalable
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
What do we get?Collecting job execution time statisticsAverage job execution timeStandard deviationFit a distributionDetect and avoid black holesFor normal distribution:99.7% of job execution times should lie between (avg-3*stdev) and (avg+3*stdev)96% of job execution times should lie between (avg-2*stdev) and (avg+2*stdev)
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Detecting hanging transfers
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Setting Execution Time LimitsAvg = 7.8 minStdev = 3.17minFor normal distribution:%99.7 : [0 17.31 min]%96 : [1.46 min 14.14 min]
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
What do we get? (2)Identifying misconfigured machinese.g. find set of machines which fail jobs with I/O data size larger than 2 GB (i.e. OS limitations)Identifying factors affecting job run-timeBug huntingJob failures on certain inputs Memory leaksScheduler logs image size regularly
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Catching Memory LeaksJob Memory Image Size (MB)Time
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
What do we get? (3)Application optimizationHow long does each step of an application/pipeline take to execute?AdaptationFind resources that take least time to execute jobs from a particular class
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
ConclusionsView of the Grid from the client sideJob/user log files as main source of informationAggregate experience of different jobs and pass them to future onesHelps in:Catching black holesIdentify faulty/misconfigured resourcesBug trackingStatistics collectionFuture work: Merge experience of different clients
A Client-centric Grid KnowledgebaseGeorge Kola, Tevfik Kosar and Miron Livny
Thank you
For more information, contact:
Tevfik Kosarhttp://www.cs.wisc.edu/[email protected]