April 21, 20231

GrenchMark:Synthetic workloads for Grids

First Demo at TU Delft

A. Iosup, D.H.J. EpemaPDS Group, ST/EWI, TU Delft

April 21, 20232

Evaluating Grid schedulers performance

• Grid schedulers performance• Qualitative metrics

supported application types, advanced fault tolerance, advanced ...

• Quantitative metricsresource consumption, system performance, success rate

• Other metricscost, single number system description

• Needs• Applications, workloads, more workloads…

April 21, 20233

Synthetic workloads for Grid schedulers

• Good synthetic workloads for Grid schedulers• Specific scheduler comparison requirements

(metrics, jobs inter-arrival time, ...)• Many different types of representative Grid applications• Traditional software engineering requirements

(flexibility, extensibility, usability, ...)

• Can also be used for... • Functionality testing and system tuning• Application performance testing• Systems design and procurement• …

April 21, 20234

Representative Grid applications

• Unitary applications• Just one scheduling unit (otherwise recursive

definition)• Examples:

Sequential, MPI, Java RMI, Ibis, …

• Composite applications• Composed of several unitary or composite applications• Examples:

Parameter sweeps, chains of tasks, DAGs, workflows, …

April 21, 20235

Outline

• Introduction• The GrenchMark framework• Experience with GrenchMark • Extending GrenchMark• Conclusions

[done]

[here]

April 21, 20236

The GrenchMark framework

• What’s in a name?grid benchmark → help standardizing the testing procedures, but benchmarks are too early…

• GrenchMarkA systematic approach to testing Grid schedulers• A set of metrics for comparing schedulers• A set of representative Grid applications

• Both real and synthetic• Easy-to-use tools to create synthetic workloads• Flexible, portable, extensible

• Can also be used for testing other Grid components

April 21, 20237

The GrenchMark framework

April 21, 20238

GrenchMark: Preliminary notions

• Job, workload, workload unit• Job = Set of components (support for co-allocation)

[Job = one program execution / the basic scheduling unit / … ] • Workload = Set of Jobs• Workload Unit = Set of jobs with the same

property, generated from one description line (definition useful only for workload generator)

• Other• JDF = Job description file• Inter arrival time

April 21, 20239

GrenchMark: Some more notions

• Job site type / per site description• Single – run at one site• Fixed – run at several sites, all specified

my license is on those machines

• Un-Fixed – run at several sites, all unspecified I can run anywhere, just give me the resources

• Semi-fixed – run at several sites, some specified I prefer those machines, but I can work anywhere

• Inter-arrival time distributions Constant, Uniform, Normal, Exponential(λ), Poisson(Mean), HExp2, HPoisson2, Weibull, LogNormal, Gamma (~)

April 21, 202310

WL description: an example

• Describe the workload to be generated in a few lines

• Very simple language + custom extensions (Native) (ExternalFile field)

• Support• Co-allocation• Start time, inter-arrival time• Mixes of jobs

April 21, 202311

The GrenchMark process

$ ./wl-gen.py wl-desc.in $ ./wl-submit.py out/wl-to-submit.wl

12

1

2

Semi-automated

• Sample run:• 4 lines of description• 100 jobs / 411 components• 100 files / 132 directories• 300KB data

• Sample run:• defined inter-arrival rate – submission delay +/- 0.01s• 100 JDFs

April 21, 202312

GrenchMark status

• Already done in Python [http://www.python.org]• Generator + Globus, KOALA generators + RSL printer

• Submitter• Results analyzer (crude)

• Applications• Unitary, 3 types: sequential, MPI, Ibis (Java)• +35 different applications

• Ongoing work• Composite applications• Automated results analyzer

April 21, 202313

Demo:Generating mixes of jobs

• 10 jobs• 8 MPI, multi-component jobs (need co-

allocation)• 2 sequential

April 21, 202314

Outline

• Introduction• The GrenchMark framework• Experience with GrenchMark • Extending GrenchMark• Conclusions

[done]

[here]

[done]

April 21, 202315

GrenchMark for testing KOALA…

• Testing• 3 different runners: drunner, grunner, krunner• Pre-release status: supposed stable• Workloads with different jobs requirements, inter-

arrival rates, co-allocated v. single site jobs…• Evaluate• Jobs success rate, KOALA’s overhead and

bottlenecks• Results• +5,000 jobs successfully run• 2 major bugs first day, +10 bugs overall (all fixed)• KOALA is officially released

(full credit to KOALA developers, 10x for testing with GrenchMark)

April 21, 202316

GrenchMark for testing KOALA: A full workload example

• KOALA test workload, run 10 times:• Globus MPICH-G2 / MPI jobs• Components: 4 and 8• Component Sizes: 4, 8 and 16• Inter-arrival time: Poisson(5s), spikes• Co-allocation, 1 site• Submit time 1 day

• Generate:

• Submit:

• Total: 3200 jobs, 19200 components / 3k files, 4k dirs• Timing: 30s generate / 86,400s submit (1 day)

$ wl-gen.py --duration=86400000 wl-desc.in

$ wl-submit.py –onefile out/wl-desc.in

April 21, 202317

… and DAS-2’s functionality• Already done

• Evaluate for KOALA + Globus + DAS-2• jobs success rate, turnaround time, middleware overhead,

types and sources of errors• Results

• 5 workloads• 500 jobs

A.Iosup, J.Maassen, R.V.van Nieuwpoort, D.H.J.Epema, Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark, 2005 (submitted).

• Currently• examine DAS-2 support for composite applications

April 21, 202318

Outline

• Introduction• The GrenchMark framework• Experience with GrenchMark • Extending GrenchMark• Conclusions

[done]

[here]

[done]

[done]

April 21, 202319

Extending GrenchMark (1)

• Motto:Extending GrenchMark is easy!

• Need:• Good knowledge about the application

type• Good understanding of workflows, or

Good understanding of grid middleware• Minimal Python knowledge

(follow the official 2hrs tutorial: http://docs.python.org/tut/tut.html)

April 21, 202320

Extending GrenchMark (2)

1. Write your own Job Generators• a function with a predefined name in a Python module• auto-loaded

2. Write your own Unit Generators• a function with a predefined name in a Python module• auto-loaded

3. Interface with C/C++, Ruby, Perl, Java, …• define your own protocol

4. Write your own printers• a function with a predefined name in a Python module• auto-loaded

April 21, 202321

Outline

• Introduction• The GrenchMark framework• Experience with GrenchMark • Extending GrenchMark• Conclusions

[done]

[here]

[done]

[done]

[done]

April 21, 202322

Conclusions and future work

• GrenchMark generates diverse workloads of Grid applicationseasy-to-use, flexible, portable, extensible, …

• Experienceused GrenchMark to test KOALA’s functionality and performance. used GrenchMark to test some DAS Grid functionality. +5,000 jobs generated and run … and counting.

• (more) advertismentHave specific Grid applications types you would like to test? Test with GrenchMark!

April 21, 202323

Thank you!

Questions? Remarks? Observations? All welcome!

Grenchmark http://grenchmark.st.ewi.tudelft.nl/ [10x Paulo]

Alexandru IOSUPTU Delft

A.Iosup@ewi.tudelft.nl

http://www.pds.ewi.tudelft.nl/~iosup/index.html [google: “iosup”]