parallel file system benchmarking project tracey wilson dice, program manager csc [email protected]
TRANSCRIPT
DICE OverviewWhat is the DICE TAP?The File System Comparison Issue and Why?DICE Plan for FrameworkNormalizationSummary
Outline
Nationwide Test Environment for Technology Innovations
Testing & Validation ServicesIndependent 3rd Party Evaluations
Evaluating HPC, Storage, Networking andData Management Technologies
Product & Technology Evaluation & Validations
Proof-of-Concept & Performance Validations
Technology Research & Development Projects
Benchmark & Standards Development & Validation
DICE utilizes a geographically distributed test bedReal world research networksDICE sites indicative of HPC Data Centers
DICE Architecture
Quite Simply…A Test Bed for Advancing Innovation
Independent, Unbiased 3rd Party Evaluations
Genesis of a Vision by Government, Industry & Academia to Address Critical HPC Technology Challenges
The DICE Technical Advisory PanelAdvises DICE on:
Technology directionsProject conceptsTesting methodologies
Utilizes working groups to focus on more specific issuesCurrent members include:
Community Input is Key
Kevin Regimbal, PNNL Chris Jordan, TACC
Lee Ward, Sandia Steve Conway, IDC
Kevin Wilson, P&G Dan Duffy, NASA
Ron Bewtra, NOAA Lloyd Slonaker, DoD
Matt Leininger, LLNL
Lack of standardized metrics for performanceNeed exists for file system performance comparisonNeed exists for defined benchmarks for local and remote file systems Different HPC systems and their file system architectures have no direct correlations
Scaling:Scaling of current file systems has unpredictable performance impacts
Load:Performance varies with file systems with loads of 50-70%Fragmented data is not accounted for
What is the File System Issue?
Centers need input for:New procurementsUser requirementsApplication performance
File systems are a critical piece of an HPC solutionHow do you as a Center Director make an informed decision?
Why do HPC Centers Care?
“Can you prove or rather disprove the performance or value of one file system compared to another?”
In 2007, DICE received 3 proposals for evaluation projects:
PanasasDoDMicrosoft
DICE Advisory Panel offered new solution“Develop a standard benchmark framework for file
system evaluation”
How did the Effort Begin?
1. Make the framework expandable to use 2. Make it fully available for the HPC community to use3. Effort was to include normalization of results
Benefits of Standardization
Standardized MetricOne set of benchmarks for all parallel file systemsResults will be normalized for comparisonHPC Community needs non-biased benchmark(s)
Increased understanding of the impacts on current and proposed upgrades to a center’s file system:
The storage subsystem performance ScalingLoad performance
Current Ideas
Parameterized Benchmark SuiteSimulate many different workloadsExpose more tuning/configuration options in the benchmarksSuite Extensible so users can add new benchmarksHave unique benchmarks for synthetic workloads
TransactionsStreamingRandom I/OVarious Read/Write Ratios
What is the plan?
Currently surveying the HPC benchmark and file system community including:
Carnegie MelonUniversity of California/Santa CruzORNLArgonne National LabsPNNLTACCLLNLAnd many more
Application I/O Patterns
Many different I/O intensive applicationsInvestigate the development of an I/O analyzer
Recognize I/O utilization of applicationsSimilar to trace and trace replay
Add this functionality to the framework to simulate application I/O patternsVerification of performance
Can be applied on different architectures
Normalization
Normalization is needed between differing architectures
Need a formulaPerformanceCostAdditional Criteria will be evaluated
Aging and fragmentation of data should be consideredDifficult to develop
Need to develop on a small scale firstWill the normalization hold as you scale systems?
Plan is to create the framework first then normalize
Summary
Lot of collaborative effort from the Team and the Community is requiredFull project plan is in draftSurvey underway to identify current effortsPlan to have initial framework for file system testing by early 2010Plan to use this on other file system types
How To Get Involved
Collaboration is keyDeveloping a DICE Forum for this projectWill be collecting tracesDICE TAP will be forming specific working group for this effortInterested parties should send email to [email protected]
DICE Website www.diceprogram.org