data analysis and visualization of “big data” from hpc · 2013-03-28 · programming with big...
TRANSCRIPT
![Page 1: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/1.jpg)
Sean Ahern (Univ. of Tennessee, ORNL)Dave Pugmire (ORNL)George Ostrouchov (ORNL, UT)Scott Klasky (ORNL, UT)
27 March 2013
Data Analysis and Visualization of“Big Data” from HPC
![Page 2: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/2.jpg)
What is “Big Data”?
• Big data is…
• Large: Beyond the capabilities of commonly used techniques
• HPC has been doing this for decades:• Using datasets from large-scale simulation of physical phenomena.
Extracting meaning from large sets of digital information
HPC has been doing this for decades:Using datasets from large-scale simulation of physical phenomena.Using datasets from large-scale simulation of physical phenomena.
![Page 3: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/3.jpg)
Data may be the “fourth pillar” of science
Empi
rical
Theo
retic
al
Com
puta
tiona
l
![Page 4: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/4.jpg)
Data may be the “fourth pillar” of science
Dat
a-dr
iven
Empi
rical
Theo
retic
al
Com
puta
tiona
l
![Page 5: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/5.jpg)
So what’s all the hype about?
• Much of the hype of the last two years has been about extracting meaning…• …from large unstructured and semi-structured datasets
• …from the Business Intelligence community
• …using scalable analysis and storage tools like:Hadoop, Cassandra, HBase, Hive, Pig, MongoDB, …
• Some successes in scientific computing for unstructured data
![Page 6: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/6.jpg)
Petascale machines are solving important scientific problems through the generation and analysis of large datasets
Physics of high-temperature superconducting cuprates
Chemical structure of HIV viral envelope
Global climate simulation of CO2dynamics and biochemistry
Biological conversion of cellulosic feedstock for biofuels
Three-dimensional simulation of blood flow
Combustion of turbulent lean fuel mixtures
![Page 7: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/7.jpg)
Current success comes from having tools that scale to the HPC datasets
• Analysis methods are generally the same as in serial
• It’s the tools and techniques that must scale
• SPMD data parallelism has taken us far -> Petascale+
I/ODecomposed
data setParallelAnalysis
Code
Rendering/compositing
0
12
33
2
1
0 Proc 0
Proc 1
Proc 2
![Page 8: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/8.jpg)
We’ve scaled to trillions of mesh cells
2T cells, 32K procs
Machine Type Problem Size # coresJaguarFranklinDawnJunoRangerPurple
Cray XT5 2T 32kCray XT4 1T, 2T 16k, 32k
BG/P 4T 64kLinux 1T 16k, 32k
Sun Linux 1T 16kAIX 0.5T 8k
Since this work, people havereached 8 trillion cells = 20,0003 cells.
• Weak scaling study 2009 (isocontouring, volume rendering): ~63 million cells/core
![Page 9: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/9.jpg)
The “Three V’s” of big data as applied to HPC
BigData
Volume Variety
Velocity
![Page 10: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/10.jpg)
The “Three V’s” of big data as applied to HPC
• Volume:• Increasing mesh resolution
• Increasing temporal complexity
• Variety:• Higher-dimensional data
• Increasing multivariance
• Complex data models
• Ensembles, parameter studies
• Velocity:• Future hardware constraints will limit I/O
Volume
Velocity
Variety
High-dimensional
Real-time
In situ
Post-processing
Ensembles
3D grids
Coupledexperimental
2D grids
Hyperspectral
Energygroups
Staginganalytics
Experimentaldata
![Page 11: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/11.jpg)
Data scale limits scientific understanding
• Spatial resolution increasing in many domains• Too many zones to see on screen (with or without high-res Powerwall)
• Temporal complexity increasing• Climate simulations will have 100,000s of time steps
• Manually finding temporal patterns is tedious and error-prone
• Multivariate overload• Climate simulations have 100-200 variables
• Growing to thousands
• Issues of data models and domain-specific data• E.g., Multigroup radiation fields
![Page 12: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/12.jpg)
Common Themes in Mathematics for Data
Computation Mode
Memory Architecture
Data PartitionAvailability
Data Passes
Serial or Multithreaded
contiguous N/A (all) many
Distributed partitioned all many
Out-of-Core partitioned one-at-a-time one or few
Streaming/in situ
partitioned one-at-a-time one
Mathematics needed
partition coupling math
updating via partition coupling math
one-pass updating via partition coupling math
Minimizing data access (energy) pushes everyone toward one-pass (one-touch) updating.
![Page 13: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/13.jpg)
Common Themes in Mathematics for DataComputation
ModeMemory
ArchitectureData Partition
AvailabilityData Passes
Serial or Multithreaded contiguous N/A (all) many
Distributed partitioned all many
Out-of-Core partitioned one-at-a-time one or few
Streaming/in situ partitioned one-at-a-time one
Mathematics needed
partition coupling math
updating via partition coupling math
one-pass updating via partition coupling math
Minimizing data access (energy) pushes everyone toward one-pass (one-touch) updating.
Distributed many
Out-of-Core
Streaming/in situ
Streaming/
![Page 14: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/14.jpg)
Prediction of Ice and Climate Evolution at Extreme Scales (PISCEES)
• SciDAC Project that is: • Developing robust, accurate, and scalable
dynamical cores for ice sheet modeling on a variety of mesh types
• Evaluate ice sheet models using new tools for V&V and UQ
• Integrate these models and tools into the Community Ice Sheet Model (CISM) and Community Earth System Model (CESM)
• Simulate decade-to-century-scale evolution of the Greenland and Antarctic ice sheets
PI: William Lipscomb, Philip Jones (Acting)LANL
![Page 15: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/15.jpg)
Data set structure
• Structured NetCDF files• Ice defined by two variables: topg, thk
• Ice flow sheets defined by N levels
land height
ice thickness
Ice sheet mesh
![Page 16: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/16.jpg)
Flow Analysis in PISCEES
Sea Level
10K feet
![Page 17: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/17.jpg)
We have a scalable integral curve algorithm Parallelize over blocks
Parallelize over seeds
• Use a combination of seed and block parallelization• Dynamically steer computation
• Allocate resources where needed
• Load blocks from disk on demand
• Parallelize over seeds and blocks
• Goals:1.Maximize processor utilization
2.Minimizes IO
3.Minimizes communication
• Works adequately in all use casesD. Pugmire, H. Childs, C. Garth, S. Ahern, G. Weber, “Scalable Computation of Streamlines on Very Large Datasets,” Proceedings of Supercomputing 2009, Portland, OR, November, 2009
![Page 18: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/18.jpg)
Parallel Integral Curve System (PICS)
Data flow network
Disk
PipelineFilter Filter Filter Render
![Page 19: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/19.jpg)
Parallel Integral Curve System (PICS)
Data flow network
Disk
PipelineFilter Filter Filter Render
PICS Filter
StopTerminate?
Step
Analyze
Advect
SeedsField
![Page 20: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/20.jpg)
PICS Filter
StopTerminate?
Step
Analyze
Advect
SeedsField
PICS Filter Design
Seeds
Instantiate particles
Step
Advect
Advance particles
Field
Evaluate velocity
Terminate?
AnalyzeAnalyzeAnalyzeAnalyzeAnalyzeAnalyzeAnalyzeAnalyzeAnalyze
Analyze step
Stop
Create output
Parallelization
![Page 21: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/21.jpg)
“Distance Traveled” operator
![Page 22: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/22.jpg)
“Distance Traveled” operator
12.83
![Page 23: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/23.jpg)
Results
Steady state flow distance traveled
1 year of travel 10 years of travel
![Page 24: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/24.jpg)
Big Data Exploration Requires Flexibility in Analytics
• Analytics for not-so-big data:• Science produces great many
algorithms
• Often redundant and related
• Statistics and mathematics prune, generalize, and connect
• Still we have hundreds of statistics and mathematics books on data analytics
No reason to expect big data less needy than not-so-big dataFlexibility and a big toolbox are critical for exploration
Need for analytics diversity persists for big data
![Page 25: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/25.jpg)
Programming with Big Data in R: pbdR
• R has unmatched diversity and flexibility for data
• Goals:• Shorten time from big data to science insight
• Productivity, Portability, Performance
• Approach:• Implicit management of distributed data details
• Scalable, big data analytics with high-level syntax
• Identical syntax to serial R
• Powered by state of the art scalable libraries
• Free* R packages
*MPL, GPL and BSD licensed
pbdR Core TeamGeorge Ostrouchov1,2, Team LeadWei-Chen Chen1
Pragnesh Patel2Drew Schmidt2
1Oak Ridge National Laboratory 2University of Tennessee
![Page 26: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/26.jpg)
High-Level Syntax
Cov.X <- cov(X) Cov.X <- cov(X)
N <- nrow(X)
mu <- colSums(X) / N
X <- sweep(X, STATS=mu, MARGIN=2)
Cov.X <- crossprod(X.spmd) / (N-1)
N <- allreduce(nrow(X))
mu <- allreduce(colSums(X) / N)
X <- sweep(X, STATS=mu, MARGIN=2)
Cov.X <- allreduce(crossprod(X)) / (N-1)
Covariance
Dev
elop
er
Linear Models
Lm.X <- lm.fit(X, Y) Lm.X <- lm.fit(X, Y)
tX <- t(X)
A <- tX %*% X
B <- tX %*% Y
ols <- solve(A) %*% B
tX <- t(X)
A <- allreduce(tX %*% X)
B <- allreduce(tX %*% Y)
ols <- solve(A) %*% BDev
elop
er
![Page 27: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/27.jpg)
A High-Level Language for Big Data Analytics
pbdBASE(
pbdDMAT(
OpenMPI,))MPICH)
))))))
MPI)
ACML,)MKL,)libsci)))))
LAPACK)
BLAS)
MKL,)libsci))))))
ScaLAPACK)
PBLAS)
BLACS)
pbdSLAP(pbdNCDF4(
ADIOS,)NetCDF4,)HDF5)
))))))
Parallel)I/O)
pbdR(r?pbd.org)
High(Performance(Libraries(
pbdMPI(
pbdADIOS(
pbdDEMO(
![Page 28: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/28.jpg)
A High-Level Language for Big Data Analytics
pbdBASE(
pbdDMAT(
OpenMPI,))MPICH)
))))))
MPI)
ACML,)MKL,)libsci)))))
LAPACK)
BLAS)
MKL,)libsci))))))
ScaLAPACK)
PBLAS)
BLACS)
pbdSLAP(pbdNCDF4(
ADIOS,)NetCDF4,)HDF5)
))))))
Parallel)I/O)
pbdR(r?pbd.org)
High(Performance(Libraries(
pbdMPI(
pbdADIOS(
pbdDEMO( Distributed Matrix Class
Built on standard parallel libraries
Managed block
distribution
Parallel data
access
![Page 29: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/29.jpg)
Promising Scalability
2
4
6
8
10
●
●
●
●
●
●
●
●
●
●
2 4 6 8 10 12Thousands of Cores
Log
Scal
ing
Rel
ative
to 1
024
Cor
es
Matrix Size●
●
100,000 x 100,00050,000 x 50,000
log−log Scale PCA Benchmark
Nautilus Kraken
0
200
400
600
64 128 256 512Cores
Mea
n R
un T
ime
(sec
onds
) fro
m 2
5 R
uns
Construct Covariance
●
●
●
●
2
4
6
8
64 128 256 512Cores
Aver
age
Scal
ing
(rela
tive
to 6
4 co
res)
from
25
Run
s
Matrix for a 750,000x5200 Matrix
![Page 30: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/30.jpg)
Promising Scalability
● ●
●
●
●
●
●
●
2.5
5.0
7.5
10.0
12.5
64 96 128 192 256 384 512 768Cores
Scal
ing
Rel
ative
to 6
4 C
ores
Scaling of lm.fit(x,y)dim(x) = 2,000,000x5000, dim(y) = 2,000,000x1
Nautilus
0.0
2.5
5.0
7.5
10.0
64 96 128 192 256 384 512 768Cores
Dat
a G
ener
atio
n (G
B pe
r Sec
ond)
Gaussian data generation
![Page 31: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/31.jpg)
Performance is highly dependent upon I/O rates
2T cells, 32K procs
• Weak scaling study 2009 (isocontouring, volume rendering): ~63 million cells/core
•Appx I/O time: 2-5 minutes•Appx processing time: 10 seconds
Machine Type Problem Size # coresJaguarFranklinDawnJunoRangerPurple
Cray XT5 2T 32kCray XT4 1T, 2T 16k, 32k
BG/P 4T 64kLinux 1T 16k, 32k
Sun Linux 1T 16kAIX 0.5T 8k
![Page 32: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/32.jpg)
Performance is highly dependent upon I/O rates
2T cells, 32K procs
• Weak scaling study 2009 (isocontouring, volume rendering): ~63 million cells/core
•Appx I/O time: 2-5 minutes•Appx processing time: 10 seconds
Machine Type Problem Size # coresJaguarFranklinDawnJunoRangerPurple
Cray XT5 2T 32kCray XT4 1T, 2T 16k, 32k
BG/P 4T 64kLinux 1T 16k, 32k
Sun Linux 1T 16kAIX 0.5T 8k
MachineJaguarFranklin
RangerPurple AIX 0.5T 8k
�0 ��0 ��0 ��0 ��0 ���0
����������������
���������������������������
������������������������� �����������������������
��� ������� ��������� ��� ��������
![Page 33: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/33.jpg)
Hardware technology curves at the exascale: ~2018
• Billion-way concurrency
• Constrained memory environment
• Very constrained I/O(in relation to compute capability)
NSF CIF21: “Computing power, data volumes, software, and network capacities are all on exponential growth paths.”
![Page 34: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/34.jpg)
Hardware technology curves at the exascale: ~2018
• Billion-way concurrency
• Constrained memory environment
• Very constrained I/O(in relation to compute capability)
NSF CIF21: “Computing power, data volumes, software, and network capacities are all on exponential growth paths.”
![Page 35: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/35.jpg)
Implications for analysis & visualization
• Extremely limited ability to move data off HPC resource
• The de-coupling of simulation codes and visualization codes can no longer continue
• Extremely limited memory space
• Extreme concurrency will break most communication models
• Pure MPI already breaking at the petascale
• Fine degree of data parallelism will break most data processing models
![Page 36: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/36.jpg)
So what do we do?
• We’ve had a lot of success moving processing closer and closer to the data.
• Medium-term goal: Take the final step in moving processing closer to the data• Right to the source, in memory where it’s generated
• Reduce data every step of the way to the user
• All data visualization (and analysis) becomes remote
• Long-term goal: Move data processing everywhere the data lives
![Page 37: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/37.jpg)
Injection of analysis into I/O pipelines
• Leverages existing code methods for data movement• Lots of compute available
• Exploits high-capacity HPC interconnect
• Possible to annotate, reduce, and reorder data in flight
• I/O is the primary bottleneck, so it’s the logical place to begin adaptation
• Challenges:• Memory-constrained
• Impossible to do serendipitous discovery with in situ alone
• Temporal analysis can be difficult
![Page 38: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/38.jpg)
Goal: Tightly-coupled analysis infrastructures
In situ processing*
Post processing
Simulation Simulation
Analysis
Visualization
Downstream tools
AnalysisVisualization
Downstream tools
These codes will be designed to runin situ on the same node as the simulation or on a tightly coupled co-processing node
These codes will be run as a post-process. They do not currently scale for in situ processing.
E.g., S3D, GTCGenerate raw data.
Feedback intosimulation:e.g., regions forparticle injection
Analysis/Visresults
E.g., Integrated volume, surface,and particle rendering• Render raw data or segmented features.• Render particles and trajectories
E.g., Statistical analysis offeatures, PCA• Generate distributions, low- dimensional linear representations.
E.g., Merge tree• Segment data and extract features of interest.• Query particles and track features.
E.g., MS-Complex• Partition space using gradient flow.• Cluster particles and trajectories.
E.g., Integrated volume, surface,and particle rendering• Render MS-complex.• Render particles and trajectories clusters.
E.g., Isomap• Generate low-dimensional non- linear representations.
* Code for in situ processing can also be used in post processing
CUDA/OpenCL
OpenMP + MPIDisplayStorage
Topology fileformat
Simulation, analysis, visualization workflow
Time t Time t+∆t
MPI
MPI
MPI
MPI
MPI
MPI
MPI
MPI
GPU
GPU
GPU
GPU
GPU
MPI
MPI
GPU
Topology fileformat
Data transfer:In memory orvia ADIOS
Simulation
Coupled analysis
Post processing
Infra
stru
ctur
e by
Jac
kie C
hen
of th
e Sa
ndia
Com
bust
ion
Cent
er, 2
011
![Page 39: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/39.jpg)
ADIOS: ADaptable I/O System
• An I/O abstraction framework
• Provides portable, fast, scalable, easy-to-use, metadata rich output with a simple API
• Layered software architecture
• Change I/O method on-the-fly
• Abstracts the API from the method used for I/O
• Provides method for injection of analysis and visualization
Interface)to)apps)for)descrip/on)of)data)(ADIOS,)etc.))
Buffering)Feedback) )Schedule)
Mul/Bresolu/on)methods)
Data)Compression)methods)
Data)Indexing)(FastBit))methods)
Data)Management)Services)
Workflow))Engine) Run/me)engine) Data)movement)Provenance)
Plugins)to)the)hybrid)staging)area)
Visualiza/on)Plugins)Analysis)Plugins)
Parallel)and)Distributed)File)System)
IDX) HDF5)AdiosBbp) pnetcdf) “raw”)data) Image)data)
Viz.)Client)
ADIOS Performance
HDF5 Performance
![Page 40: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/40.jpg)
ADIOS: I/O problems and their solutions
• I/O buffering and aggregation:• Buffering utilizes max I/O bandwidth at writing
• Aggregation utilizes max I/O bandwidth at reading
• Data layout optimization:• Layout optimization improves read performance
• Reduce synchronization and contention:• Log-file format and aggregation decreases lock
contention and decrease network movement
• Scheduled staging decreases impact of asynchronous I/O
5% peak
0.8% peak
![Page 41: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/41.jpg)
Success across many application domains
• Fusion:• Using staging towards predictive capabilities
• Coupled with ParaView, VisIt, and dashboards
• Combustion:• Orders of magnitude speed improvement for
checkpoint/restart
• In situ feature detection
• More (astro, AMR,geophysics, …)
Pixie3D MHD Fusion simulation
Pixplot Analysis code
ParaView parallel server
Analysis code record.bp record.bp record.bp
pixie3d.bp pixie3d.bp pixie3d.bp pixie3d.bp DataSpaces
![Page 42: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/42.jpg)
At the same time: Exploit fine-grained parallelism
• Memory space per core may be down in the megabytes to 100s of kilobyte range• (Though HMC’s may alleviate the memory wall and capacity.)
• Data parallelism becomes much more complex at this fine level of detail:• How to create and exploit search data structures
• How to harness communication hierarchies
• How to handle “ghost” region between data blocks
• Iterative algorithms (e.g., streamlines) become extremely inefficient
• No analysis or visualization algorithms are written this way.Everything must be rewritten.
![Page 43: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/43.jpg)
Extreme-scale Analysis and Visualization Library: EAVL
• Update traditional data model to handle modern simulation codes and a wider range of data. Extend current analysis and vis tools to data that cannot be approached today.
• Investigate how an updated data and execution model can achieve the necessary computational, I/O, and memory efficiency.
• Explore methods for visualization algorithm developers to achieve these efficiency gains and better support exascale architectures. Allow easy gradual adoption into existing analysis/vis tools.
![Page 44: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/44.jpg)
Three primary activities
• Provide a richer data model for scientific data expression
• Provide new execution models for constrained memory, advanced architectures• Distributed parallelism, task parallelism,
thread parallelism
• Provide programming method to exploit the above: functor + iterator, library of methods
������
�����
��
��� �����������������
�
�
�
�
�
�� � �
�
Provide new execution models for
![Page 45: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/45.jpg)
Promising results to date
• Data thresholding is more memory- and computation-efficient.
• Coordinate space transformation moves from O(n) to O(1).
• Structured grids are an order of magnitude more space efficient.
• Efficient execution on heterogenous architectures
• Temporal parallelism provides scalability benefits
• In-place processing provides significant memory benefits
���
���
����
����
�� ��� ����
������������
����
����
��
��������� ������
��� ����
�����
�����
�����
�����
�����
� � �� �� ���
������
����
��������������
� �� �����������������
������������������ �������������������������������� ��������������
![Page 46: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/46.jpg)
HPC data is becoming more complex
• Workflow tools are becoming more common:• Pegasus, Swift, HTCondor, Kepler, Eden, …
• Ensembles are growing larger
• Parameter studies and Design of Experiments are becoming much more common
• Tracking provenance is important
• There are no general tools for processing large runs of simulations
The concept of a “data set” is evolving…becoming more unstructured.
![Page 47: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/47.jpg)
40
Turbo Compressor InnovationBreakthrough aerodynamic design optimization
OLCF Contribution Results
Science Objectives and Impact
• Ramgen Power Systems is developing shock wave compression turbo machinery to meet DOE goals for reducing Carbon Capture and Sequestration (CCS) costs
• Complementary goal: design a gas turbine with dramatically lower costs and higher efficiency
• Compressing CO2 to the required 100 atmospheres represents approximately 33 percent of the total cost of Sequestration
• Transformed Ramgen’s aerodynamic design process• Observed designs with valuable new characteristics, from
ensembles not possible without Jaguar• Created a new workflow paradigm that accelerates design of
compressors• Accelerated computational design cycle for turbo machinery from
months to 8 hours!
Leadership-scale ensemble runs on Jaguar support intelligently-driven design optimization
• 50x improvement in code scalability with more efficient memory utilization
• Accelerated I/O by 10x with optimizations and ADIOS
• Intelligent use of ensembles to explore parameter space using 240,000 cores
"The use of Jaguar has cut the projected time from concept to a commercial product by at least two years and the cost by over $4 million,” -- Ramgen’s CEO and Director Doug Jewett.
INCITE ProjectAllan Grosvenor, Ramgen Power Systems
Allocated hours: 40MUsed hours: 36M
0%30%60%
<20% 20-60% >60%
Compute Usage
Intelligent use of ensembles to explore parameter space using 240,000 coresIntelligent use of ensembles to explore Intelligent use of ensembles to explore Intelligent use of ensembles to explore
Observed designs with valuable new characteristics, from ensembles not possible without JaguarCreated a new workflow paradigm that accelerates design of
Observed designs with valuable new characteristics, from Observed designs with valuable new characteristics, from ensembles not possible without JaguarCreated a new workflow paradigm that accelerates design of
Observed designs with valuable new characteristics, from ensembles not possible without Jaguarensembles not possible without JaguarCreated a new workflow paradigm that accelerates design of ensembles not possible without Jaguarensembles not possible without JaguarCreated a new workflow paradigm that accelerates design of compressorsCreated a new workflow paradigm that accelerates design of Created a new workflow paradigm that accelerates design of Created a new workflow paradigm that accelerates design of Created a new workflow paradigm that accelerates design of Created a new workflow paradigm that accelerates design of
![Page 48: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/48.jpg)
41
Science Objectives and Impact• Predict how many protons and neutrons can be
bound within a nucleus.• Identify the nuclear drip lines that denote the
limits of nuclear binding.• Project advances toward the vision of “designer
nuclei,” with uses ranging from potential cancer treatments to a better understanding of superconductivity.
Nuclear Physics The Limits of the Nuclear Landscape
Map of bound even–even nuclei as a function of Z and N. Shown are stable nuclei (black), radioactive nuclei (green), mean drip lines with uncertainties (red), and two-neutron separation line (blue). The inset shows the irregular behavior of the two-neutron drip line around Z = 100.
• Custom code (co-authored by OLCF staff member Hai Ah Nam) used density functional theory to solve the nuclear structure, which involved a large algebraic non-linear eigenvalue problem.
• Each ensemble of nuclei took about two hours to calculate on 224,256-processors in Jaguar system.
• Each run evaluated about 250,000 possible nuclear configurations.
OLCF Contribution Science Results
• Accurate calculation of the number of bound nuclei in nature.• Calculations predicted about 7,000 possible combinations of
protons and neutrons allowed in bound nuclei with up to 120 protons.
• Several leading models of nuclear interaction shown to be largely in agreement.
J. Erier, et al., “The Limits of the Nuclear Landscape” Nature, June 2012.
INCITE ProjectJames Vary, Iowa State University
Allocated hours: 37MUsed hours: 60M0%
35%70%
<20% 20-60% >60%
Compute Usage
problem. Each ensemble of nuclei took about two hours to calculate on 224,256-processors in Jaguar
problem. Each ensemble of nuclei took about two hours to
Each run evaluated about 250,000 possible nuclear configurations.Each run evaluated about 250,000 possible Each run evaluated about 250,000 possible nuclear configurations.
Accurate calculation of the number of bound nuclei in nature.Calculations predicted about 7,000 possible combinations of protons and neutrons allowed in bound nuclei with up to 120
Accurate calculation of the number of bound nuclei in nature.Calculations predicted about 7,000 possible combinations of protons and neutrons allowed in bound nuclei with up to 120
![Page 49: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/49.jpg)
42
Science Objectives and Impact• Determine how much sea level will rise
based on three climate-change mitigation scenarios
• Examine mechanisms involved in sea-level rise, including ice melt and thermal expansion of water
• Quantify sea-level rise in response to mitigation strategies
• Used CCSM4 to conduct simulation ensembles for each mitigation scenario
• Simulation of 15 separate solutions, 4,500 total simulation years
OLCF Contribution Science Results• Sea level will continue to rise in the future, even with aggressive CO2
mitigation scenarios.• Aggressive mitigation measures strongly affect the rate of increase• Mitigation buys important time to implement adaptation measures for
inevitable sea-level rise
CoolingStableWarming
Sea-Level Rise InevitableAggressive Greenhouse GasMitigation Can Help Slow Rate
Sea level will continue to rise due to thermal expansion into 2300 under the most aggressive mitigation scenario, (cooling), but the rise will be slowed enough to implement adaptation measures. With less aggressive mitigation (stable) and (warming), there would be less time for adaptation.
G.A. Meehl, et al., “Relative outcomes of climate change mitigation…,” Nature Climate Change, August 2012.
INCITE ProjectWarren Washington, NCAR
Allocated hours: 56MUsed hours: 49M
0%50%
100%
<20% 20-60% >60%
Compute Usage
Simulation of 15 separate solutions, 4,500 total simulation yearsSimulation of 15 separate solutions, Simulation of 15 separate solutions, Simulation of 15 separate solutions,
20-60%
INCITE ProjectWarren Washington, NCAR
Allocated hours: 56MUsed hours: 49M
INCITE ProjectWarren Washington, NCAR
INCITE ProjectWarren Washington, NCARWarren Washington, NCAR
Allocated hours: 56MUsed hours: 49M
![Page 50: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/50.jpg)
43
Science Objectives and Impact• Test millions of compounds against a specific
protein receptor using simulations.• Find the best match between a compound and its
targeted protein receptor, in hopes of creating drugs with a higher degree of specificity and less cross-reactivity, as well as exploring alternative uses for existing drugs.
• Impact: Drastically decrease the time and money spent bringing new, improved drugs to market.
Computational and Molecular BiophysicsHigh-Performance Computing for Accelerated Drug Discovery
Director’s Discretion ProjectUsage between 1/1/2012–9/30/2012Used 7.6 M hours of allocation, or 110%Used variations of AutoDock, a General Public License, open-source software, commonly used for molecular 3D modeling in protein-docking simulations.
OLCF Contribution Science Results• Successfully screened 2 million compounds against a targeted
receptor in a matter of days, as opposed to months using computing clusters or longer with test-tube methods.
• Allowed scientists to account for specific binding in protein receptors as well as structural variations within the receptor.
• Enabled search of a vast library of molecular compounds to find alternative uses for existing drugs (i.e. “repurposing”).
Computational approaches are used to describe how molecular compounds of a drug candidate (displayed in colored spheres) bind to its specific protein receptor target. Image courtesy of Jerome Baudry, UTK & ORNL
S. Ellingson. “Accelerating virtual high-throughput ligand docking,” Presented at HPDC12, 2012 .
Discretionary ProjectJerome Baudry, UTK/ORNL
Allocated hours: 7MUsed hours: 8M
0%50%
100%
<20%20-60%>60%
Compute Usage
Successfully screened 2 million compounds against a targeted receptor in a matter of days, as opposed to months using Successfully screened 2 million compounds against a targeted Successfully screened 2 million compounds against a targeted receptor in a matter of days, as opposed to months using receptor in a matter of days, as opposed to months using
![Page 51: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/51.jpg)
Southern California Earthquake Center (SCEC)
San Onofre
1. Hazard map3. Hazarddisaggregation
4. Rupture model
5. Seismograms
2. Hazard curves
CyberShake workflow
A single large HPC simulation leads to millions of serial analysis jobs
![Page 52: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/52.jpg)
What does the future hold for big data from HPC?
• Evolving nature of data sets means new research and development directions. We have no general tools to handle these!
• Workflow tools, ensemble processing, and parameter studies ->
• Need to store “meta” information about simulation results.
• Promising: Database storage and query methods
• Scalability limitations of traditional files ->
• Need to store scientific information with new methods.
• Promising: Object stores and access methods
• Fusion of HPC data and observational data ->• Need for algorithms and methods to expand into new data types.
• New data types and descriptions ->
• Requirement for processing beyond simply distributed-parallel methods
• Promising: “business analytics” methods for unstructured data
![Page 53: Data Analysis and Visualization of “Big Data” from HPC · 2013-03-28 · Programming with Big Data in R: pbdR • R has unmatched diversity and flexibility for data • Goals:](https://reader033.vdocuments.net/reader033/viewer/2022050109/5f46aea3f5a0de2cac180780/html5/thumbnails/53.jpg)
We’ve got a lot of work to do!
Thanks for your time!