mechanisms for matchmaking and parallel high throughput computing in the condor distributed system...
Post on 14-Jan-2016
220 Views
Preview:
TRANSCRIPT
Mechanisms for Matchmaking and Parallel High Throughput Computing
in the Condor Distributed System
Rajesh Raman, raman@cs.wisc.edu
Todd Tannenbaum, tannenba@cs.wisc.edu
http://www.cs.wisc.edu/condorOct 27, 1997
Condor ProjectCondor Project
OverviewOverview• What is Condor ?What is Condor ?• Projects and CollaborationsProjects and Collaborations• High Throughput ComputingHigh Throughput Computing• ClassAds and MatchMakingClassAds and MatchMaking• Parallel Computing with CondorParallel Computing with Condor
What is Condor ?What is Condor ?
High Throughput ComputingHigh Throughput Computing Distributed ResourcesDistributed Resources
• Physically distributedPhysically distributed• Distributed ownershipDistributed ownership
Resource ManagementResource Management• Increase utilization of resourcesIncrease utilization of resources• Simple interface to execution environmentSimple interface to execution environment
– User level interfaceUser level interface– Application level interfaceApplication level interface
Important MechanismsImportant Mechanisms
MatchmakingMatchmaking Checkpointing (and migration)Checkpointing (and migration)
• Owner policies require resource reclamationOwner policies require resource reclamation• Need to save (resumable) state of application Need to save (resumable) state of application
Remote System CallsRemote System Calls• Preserves submission environment in Preserves submission environment in
execution environment.execution environment. SandboxingSandboxing
• Security concernsSecurity concerns
The Condor TeamThe Condor Team
Prof. Miron Livny, PIProf. Miron Livny, PI Research StaffResearch Staff
• Todd TannenbaumTodd Tannenbaum• Derek WrightDerek Wright• Adding 2 more...Adding 2 more...
Condor Team, cont.Condor Team, cont.
Graduate Students Graduate Students • Rajesh Raman (MatchMaking)Rajesh Raman (MatchMaking)• Jim Basney (Split Execution)Jim Basney (Split Execution)• Shrinivas Ashwin (Mr. Parallel)Shrinivas Ashwin (Mr. Parallel)• Adiel Yoaz (Accounting)Adiel Yoaz (Accounting)
Undergraduate StudentsUndergraduate Students• Tom StanisTom Stanis
Condor AlmuniCondor Almuni
• Mike LitzkowMike Litzkow• David DewittDavid Dewitt• Marvin SolomonMarvin Solomon• Many others… (Produced XXX Masters and Many others… (Produced XXX Masters and
XXX PhDs]XXX PhDs]
Current Collaborators and Current Collaborators and ProjectsProjects
NCSANCSA• PACIPACI• National GridNational Grid
UW-FlockUW-Flock• Intel Sponsorship: $4.2 Million Intel Sponsorship: $4.2 Million • Graduate School, EngineeringGraduate School, Engineering
metaNEOS: metacomputing metaNEOS: metacomputing environments for optimizationenvironments for optimization• with Prof. Michael Ferriswith Prof. Michael Ferris
Condor Pool InstallationsCondor Pool Installations
UniversitesUniversites• U of Wisconsin, U of Illinois, U of Michigan, U of Wisconsin, U of Illinois, U of Michigan,
Dartmouth, Duke, U of Washington, U of Dartmouth, Duke, U of Washington, U of Virginia, U of California-BerkeleyVirginia, U of California-Berkeley
GovernmentGovernment• NCSA, Nasa, US Navy, NSA, NIKHEF NCSA, Nasa, US Navy, NSA, NIKHEF
(Amsterdam), INFN (Italy)(Amsterdam), INFN (Italy) CommercialCommercial
• Hewlett-Packard Labs, J.P. Morgan, Mercedez-Hewlett-Packard Labs, J.P. Morgan, Mercedez-Benz, Dragon SystemsBenz, Dragon Systems
Power of Computing Power of Computing EnvironmentsEnvironments
Power = Work / TimePower = Work / Time High Performance ComputingHigh Performance Computing
• Fixed amount of work; how much time?Fixed amount of work; how much time?• Response time/latency orientedResponse time/latency oriented• Traditional Performance metrics: FLOPS, MIPSTraditional Performance metrics: FLOPS, MIPS
High Throughput ComputingHigh Throughput Computing• Fixed amount of time; how much work?Fixed amount of time; how much work?• Throughput orientedThroughput oriented• Application specific performance metricsApplication specific performance metrics
Distributed Ownership of Distributed Ownership of ResourcesResources
Commodity resourcesCommodity resources• Underutilized: 70% of a pool's cycles are not Underutilized: 70% of a pool's cycles are not
utilizedutilized• Fragmented: owned by different peopleFragmented: owned by different people
Can provide HTC with these cycles, BUTCan provide HTC with these cycles, BUT• Must not impact QOS to ownerMust not impact QOS to owner
Owners specify access policyOwners specify access policy• Expressed with control expressionsExpressed with control expressions
– The current state of the resource (e.g., load average)The current state of the resource (e.g., load average)– Characteristics of the request (e.g., who wants to use it?)Characteristics of the request (e.g., who wants to use it?)– Time of day, random numbers, etcTime of day, random numbers, etc
Condor ArchitectureCondor Architecture
Startds ( Represent owners of resources)Startds ( Represent owners of resources)• Implement owner's access control policyImplement owner's access control policy
Schedds( Represent customers of the Schedds( Represent customers of the system)system)• Maintain persistent queues of resource Maintain persistent queues of resource
requestsrequests ManagerManager
• Collector: Database of resourcesCollector: Database of resources• Negotiator: MatchmakerNegotiator: Matchmaker• Accountant: Priority maintenanceAccountant: Priority maintenance
Condor Architecture, cont.Condor Architecture, cont.
MatchmakingMatchmaking
Customers Customers • Require resources with certain characteristicsRequire resources with certain characteristics• Discriminating customersDiscriminating customers• Requests place constraints on resourcesRequests place constraints on resources
Distributed ownershipDistributed ownership• Resources service requests which match owner's Resources service requests which match owner's
policypolicy• Discriminating resourcesDiscriminating resources• Resource offers place constraints on customersResource offers place constraints on customers
Matchmaking is symmetricMatchmaking is symmetric
Matchmaking with Matchmaking with Classified AdvertisementsClassified Advertisements
Parties requiring matchmaking advertiseParties requiring matchmaking advertise• Characteristics and requirements (i.e., constraints)Characteristics and requirements (i.e., constraints)
Advertisements matched by a MatchmakerAdvertisements matched by a Matchmaker Matched parties contact each other to "claim”Matched parties contact each other to "claim”
• Communication, authentication, constraint Communication, authentication, constraint verification, negotiation of terms, etc.verification, negotiation of terms, etc.
• Claiming does not involve the MatchmakerClaiming does not involve the Matchmaker Method is symmetricMethod is symmetric
• No client/server relation imposedNo client/server relation imposed
Classified Advertisement Classified Advertisement Matchmaking FrameworkMatchmaking Framework
Expression and evaluation of Expression and evaluation of characteristicscharacteristics• ClassAd, Closure, EvaluationContextClassAd, Closure, EvaluationContext
Advertising ProtocolAdvertising Protocol• Contents of advertisementsContents of advertisements• Publication protocolPublication protocol
Matchmaking AlgorithmMatchmaking Algorithm• Relates ad contents to matching process Relates ad contents to matching process • Priority schemes, Ranking schemes, etc.Priority schemes, Ranking schemes, etc.
Classified Advertisement Classified Advertisement Matchmaking Framework Matchmaking Framework (contd.)(contd.) Matchmaking ProtocolMatchmaking Protocol
• How are relevant parties informed of a How are relevant parties informed of a successful match?successful match?
• What information are they given?What information are they given? Claiming ProtocolClaiming Protocol
• How do matched parties claim each other to How do matched parties claim each other to cooperate?cooperate?
ClassAd: Mechanism for ClassAd: Mechanism for expressing characteristicsexpressing characteristics
A ClassAd is a set of names, each of which is A ClassAd is a set of names, each of which is bound to an expression. e.g., bound to an expression. e.g., [ [ Name => "Joe Hacker" ; Height => 182 ; Sex => Name => "Joe Hacker" ; Height => 182 ; Sex => "Male" ; "Male" ; Disposition => (TimeOfDay() < 600) ? "Sour" : Disposition => (TimeOfDay() < 600) ? "Sour" : UNDEFINED ;UNDEFINED ; Requirements => (other.Height < Height) && Requirements => (other.Height < Height) && (other.Sex == (other.Sex == "Female")"Female") ] ]
ExpressionsExpressions• Constants, attribute references, function callsConstants, attribute references, function calls
ClassAd (contd.)ClassAd (contd.) Attribute references may refer to attributes Attribute references may refer to attributes
in other adsin other ads• Attribute references "trigger" expression Attribute references "trigger" expression
evaluationevaluation• Scope resolutionScope resolution• Evaluates to UNDEFINED if no such expression Evaluates to UNDEFINED if no such expression
existsexists Values Values
• String, integer, real, UNDEFINED and ERROR String, integer, real, UNDEFINED and ERROR typestypes
• Operators are total (i.e., defined over all values)Operators are total (i.e., defined over all values)
Closure: Evaluation Closure: Evaluation Environment for a ClassAdEnvironment for a ClassAd
Determines which Determines which ClassAd's attributes to ClassAd's attributes to lookuplookup
Closure isClosure is• ClassAd and an ordered ClassAd and an ordered
mapping of (scope-name, mapping of (scope-name, closure) pairsclosure) pairs
• No name may be repeatedNo name may be repeated
EvaluationContext: EvaluationContext: Evaluation Environment for Evaluation Environment for several ClassAdsseveral ClassAds A set of closures which is self-containedA set of closures which is self-contained
• No closure reference leaves the contextNo closure reference leaves the context• Condor's "Standard Context" is a bit more Condor's "Standard Context" is a bit more
complexcomplex– Includes closures for a matchmaker Includes closures for a matchmaker
"advertisement”"advertisement”
Matchmaking in CondorMatchmaking in Condor
Opportunistic Resource ExploitationOpportunistic Resource Exploitation• Resource availability is unpredictableResource availability is unpredictable
– Exploit resources as soon as they are availableExploit resources as soon as they are available– Return resources as soon as they are unavailableReturn resources as soon as they are unavailable
• Matchmaking performed continuouslyMatchmaking performed continuously Attractive for malleable parallel Attractive for malleable parallel
applicationsapplications• Request more resources after execution Request more resources after execution
commencescommences– Granted immediately if resources are available, orGranted immediately if resources are available, or– As soon as resources become availableAs soon as resources become available
Matchmaking in Condor Matchmaking in Condor (contd.)(contd.)
Advertising protocolAdvertising protocol• Startd's, Schedd's send classads to CollectorStartd's, Schedd's send classads to Collector• Must contain a "Requirements” expressionMust contain a "Requirements” expression
– Optionally contain a"Rank” and “CurrentRank” Optionally contain a"Rank” and “CurrentRank” expressionsexpressions
• Startds send a "private ad" containing a Startds send a "private ad" containing a capabilitycapability
Matchmaking protocolMatchmaking protocol• Give the matched Startd and Schedd the Give the matched Startd and Schedd the
capability from the startd's private adcapability from the startd's private ad
Matchmaking in Condor Matchmaking in Condor (contd.)(contd.)
Matchmaking AlgorithmMatchmaking Algorithm• Request ad A matched with offer ad B “iff”Request ad A matched with offer ad B “iff”
– A's "Requirements" expression evaluates to TRUE, andA's "Requirements" expression evaluates to TRUE, and– B's "Requirements" expression evaluates to TRUE, andB's "Requirements" expression evaluates to TRUE, and– B‘s"Rank" expression value is greater than B‘s"Rank" expression value is greater than
"CurrentRank", and"CurrentRank", and– A’s "Rank" expression value is its greatest when A’s "Rank" expression value is its greatest when
evaluated against Bevaluated against B
Claiming protocolClaiming protocol• Negotiate "heartbeat" frequency, checkpoint Negotiate "heartbeat" frequency, checkpoint
transfer, etc.transfer, etc.
Condor ParallelismCondor Parallelism
Job LevelJob Level• Condor clusters of processesCondor clusters of processes• DagManDagMan
Task LevelTask Level• Interfacing Condor and PVMInterfacing Condor and PVM
– PVM: Message PassingPVM: Message Passing– Condor: Resource ManagementCondor: Resource Management
• PVM Resource Manager InterfacePVM Resource Manager Interface– pvm_reg_rm()pvm_reg_rm()
Interfacing Condor and Interfacing Condor and PVM, cont.PVM, cont.
Interfacing Condor and Interfacing Condor and PVM, cont.PVM, cont.
CARMI -vs- PVM CARMI -vs- PVM • Resource RequestsResource Requests
– PVM: SynchronousPVM: Synchronous– CARMI: AsynchronousCARMI: Asynchronous
• Resource Request MechanismResource Request Mechanism– PVM: Hostname and Type StringPVM: Hostname and Type String– CARMI: ClassAdCARMI: ClassAd
– CARMI Resource ClassCARMI Resource Class
• Task ManagementTask Management– CARMI: Additional NotificationsCARMI: Additional Notifications– CARMI: Additional OperationsCARMI: Additional Operations
Master-Worker ModelMaster-Worker Model A good fit for an A good fit for an
opportunistic opportunistic environmentenvironment
MasterMaster• Runs on Submit MachineRuns on Submit Machine• Manages pool of tasksManages pool of tasks
WorkerWorker• Runs on remote Runs on remote
machinesmachines• Receives pieces of work Receives pieces of work
from the Master, returns from the Master, returns answeranswer
Shadow
Master
PVMd
Starter
PVMd
Worker
Starter
PVMd
Worker
Additional Condor/PVM Additional Condor/PVM FrameworksFrameworks
CoCheckCoCheck• Checkpoint a Worker or set of WorkersCheckpoint a Worker or set of Workers• Requirements for a consistent checkpointRequirements for a consistent checkpoint
– Synchronize all processesSynchronize all processes– Flush PVM messages in transitFlush PVM messages in transit– Perform Checkpoint (save image)Perform Checkpoint (save image)– Remap TIDsRemap TIDs
WoDiWoDi• A framework for Master-Worker applicationsA framework for Master-Worker applications• Performs optimizationsPerforms optimizations
Future WorkFuture Work
DebugDebug PortPort DebugDebug PortPort DebugDebug PortPort DebugDebug Port…..Port…..
Future Work Part IIFuture Work Part II
MatchmakingMatchmaking• Aggregate Resources/RequestsAggregate Resources/Requests
AccountingAccounting• AuthenticationAuthentication
FlockingFlocking Java UniverseJava Universe Split ExecutionSplit Execution
SummarySummary
Condor is an implementation of a High Condor is an implementation of a High Throughput Computing system in an Throughput Computing system in an opportunistic environment.opportunistic environment.
Major Mechanisms to achieve HTC:Major Mechanisms to achieve HTC:• MatchmakingMatchmaking• CheckpointingCheckpointing• Remote system callsRemote system calls• SandboxingSandboxing
Questions ? Questions ?
top related