i/o behavior in hpc workflows toward understanding · toward understanding i/o behavior in hpc...
TRANSCRIPT
Toward Understanding I/O Behavior in HPC Workflows
Jakob Lüttgau, Shane Snyder, Phil Carns, Justin M. Wozniak, Julian Kunkel, Thomas Ludwig
PDSW-DISC, SC’18November 12, 2018 / Dallas, TX
Overview Motivation
Workflows & I/O Monitoring
Architecture
Demo
Outlook & Summary
Trying to add a missing link so we can movecloser to realizing smarter systems...
Require new interfaces to preserve
information about structure of data.
How to anticipate user intentions and I/O
behavior of applications ?
Require tools to observe and record system
activity as a basis to gain insight
Workflowsa HPC StoragePerspective?
Workflows offer …
… anticipatable future activity
… implicit intent to be discovered
… explicit intent description
Workflow Engines: Swift, Cylc, Tigres, etc.
Cylc, Swift-k, Fireworks
Job centric, with tasks and data targets. Tasks are
distributed and possibly run on remote systems.
Data products might be moved between sites.
Usually, a coarse granular dependency graph.
Swift-t, Tigres, Spark/RDD Lineage, QDO
A large integrated (MPI) application with many
different tasks within the application. With
exascale in mind and also closer to in situ enabled
workflows.
Closer to a programming language.
HolisticI/O Monitoringfor HPC
Tracking at the Application/Library Layer
Total Knowledge of I/O in Data Centers
Darshan: Instrumentation at Library/Application Layer
$ export LD_PRELOAD=libdarshan.so$ mpiexec -np 4 ./hellompi
[Darshan] HPC I/O Characterization Tool - https://www.mcs.anl.gov/research/projects/darshan/
TOKIO: Total Knowledge of Input/Output
Comprehensive capture of I/O activity
Support different storage services in data center
May require privileged access in many cases
[TOKIO] http://www.nersc.gov/research-and-development/tokio/
Toward Understanding Workflow I/O
Combine workflow descriptions with monitoring information from Darshan/TOKIO, etc.
Benefits:
Insight useful for operating decisions and system design
Communication with users, relatable to their scientific process
Source of information for smarter systems
Requirements:
Support multiple workflow engines as communities use different tools across difference sites
Explore convenient toolchain for researchers and operators
User facing component to communicate advice
Architecture for Augmenting I/O in Workflows
Architecture for Augmenting I/O in Workflows
Architecture for Augmenting I/O in Workflows
Architecture for Augmenting I/O in Workflows
Case Study& Demonstration
Example Workflow
Research Perspective
User Perspective
int X = 50, Y = 50;int A[][];int B[];
foreach x in [0:X-1] { foreach y in [0:Y-1] { if (check(x, y)) { // mask a region which gets computed A[x][y] = g(f(x), f(y)); // compute result for this cell (a physics process) } else { A[x][y] = 0; // default for skipped cells } } B[x] = sum(A[x]); // compute some aggregate metric}
http://swift-lang.org
https://cylc.github.io/cylc/
[scheduling] initial cycle point = 2021 final cycle point = 2023 [[dependencies]] [[[R1]]] # Initial cycle point. graph = prep => model [[[R//P1Y]]] # Yearly cycling. graph = model[-P1D] => model => post [[[R1/P0Y]]] # Final cycle point. graph = post => stop
[runtime] [[prep]] script = mpiexec -np 1 ./prep [[model]] script = mpiexec -np 4 ./model [[post]] script = mpiexec -np 1 ./post
Perspective for I/O Research and Site Operating?
Interactive Tools/Dashboards to ease navigating overwhelming amounts of log data, with “algebra”-like
semantics for convenient aggregation of multiple tasks, data objects or pipelines.
Python Library for use in, e.g., jupyter notebooks, to draft/prototype/provide templates for more
sophisticated and reproducible analysis.
JavaScript Packages (NPM) for visualisation/tools allowing easy reuse in custom tools , jupyter
notebooks (widget plugins), and dashboards (e.g., Grafana).
Communication with Scientists/Developers
Maintain affinity to scientists perspective
Stick to relationship of tasks/pipelines used by scientists/developers
Use intuitiv presentation of data-flow by extending graph of workflow
Interactive to manage complexity
100s or 1000s of different tasks and files in a workflow
Possibly, millions of log records per task (HTC, UQ)
Make it easy to aggregate multiple log records
Integration with expert advice
Human in the loop
Automatic advisories with machine learning (mid/long-term)
http://my.datacenter/workflow-io?worfklow_id=314159
What a real taskmight look like though...
Analyzing Access Patterns
Output Files
In this case diagnostic files
otherwisenot so clear
Input Files
TowardAdaptiveI/O Systems
Influence Job Scheduling decisions
Support I/O MiddlewareData Placement
Transformations
Use Case 1: I/O-Aware Scheduling for Workflows
Domain Decomposition
Raw
Data Representation
Layout on Storage
Pre/Post Out Post-Processing
Single Value: Temperature Anomaly Some average Images/Movies
CSV/Plots (x=time, y=CO2)
optimized forfast writing
Binary,optimized for transmission
optimized for fast readingor locality
Use Case 2: Benefits for I/O Middleware (1/2)
Use Case 2: Benefits for I/O Middleware (2/2)
DiscussionSummary
Requirements for Workflow Engines
Expose Context / DAGs of Workflows
Data/(file) notions
Reflection in execution runtime?
Requirements for Monitoring Solutions
Pick up context to allow associations
Support user-specific metadata with record
API to interact with monitoring toolkit
Allow counters per MPI Communicator
Requirements for Application Developers
Make intent explicit: use libs/DSL (e.g. HDF5)
Enable instrumentation with a subset of runs
Collect traces and logs for a training body.
This work was supported by the U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research,
under Contract DE-AC02-06CH11357.
Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise,
does not necessarily constitute or imply its endorsement, recommendation, or favor by the United States Government, the
Department of Energy, or the National Energy Technology Laboratory. The views and opinions of authors expressed herein
do not necessarily state or reflect those of the United States Government, the Department of Energy, or the National Energy Technology Laboratory, and shall not be used for advertising or
product endorsement purposes.
This work was supported by the ESiWACE project, which received funding from the EU Horizon 2020 research and
innovation programme under grant agreement No 675191.
The information and views set out in this work are those of the author(s) and do not necessarily reflect the official opinion of the
European Union. Neither the European Union institutions and bodies nor any person acting on their behalf may be held
responsible for the use which may be made of the information contained therein.
Disclaimer
Appendix Generic HPC Workflows
Example Climate Workflow
Common Scientific Workflows in HPCWhat makes a workflow?
SIM
UQ or HTC
in situSIM and HTC/UQ are derived figures from [1]. For outlook on workflows refer to [2].[1] LANL, NERSC, and SNL, “APEX Workflows.”, Whitepaper, Mar. 2016Online: https://www.nersc.gov/assets/apex-workflows-v2.pdf[2] E. Deelman et al., “The future of scientific workflows,” The International Journal of High Performance Computing Applications, vol. 32, no. 1, pp. 159–175, Jan. 2018.
Data-Intensive Exascale Workflow: Climate Modeling
ICON is a climate model used by Researchers at Max-Planck and by the German Weather Service (DWD).CDO is a pre/post-processing tool (climate operators) for NetCDF files.
ParaView is a popular visualisation toolkit built on top of VTK. 33/31
https://cylc.github.io/cylc/
[scheduling] initial cycle point = 2021 final cycle point = 2023 [[dependencies]] [[[R1]]] # Initial cycle point. graph = prep => model [[[R//P1Y]]] # Yearly cycling. graph = model[-P1D] => model => post [[[R1/P0Y]]] # Final cycle point. graph = post => stop
[runtime] [[prep]] script = mpiexec -np 1 ./prep [[model]] script = mpiexec -np 4 ./model [[post]] script = mpiexec -np 1 ./post