Using Kepler to Perform Parameter Using Kepler to Perform Parameter Studies in Subsurface SciencesStudies in Subsurface Sciences

Using Kepler to Perform Parameter Using Kepler to Perform Parameter Studies in Subsurface SciencesStudies in Subsurface Sciences

Jared ChaseScientific Data Management CET All Hands

Meeting 11/28/2007

http://subsurface.pnl.gov

2

Project Descriptions/GoalsProject Descriptions/GoalsProject Descriptions/GoalsProject Descriptions/Goals

CHIPPS (Tim Scheibe, Environmental Technology) Project Goal: To develop an integrated multiscale modeling

framework with the capability of directly linking different process models at continuum, pore, and sub-pore scales.

SALSSA (Karen Schuchardt, Applied Computer Science) Project Goal: To develop a process integration framework that

combines and extends leading edge technologies for process automation, data and metadata management, and large-scale data visualization.

GWACCAMOLE (Bruce Palmer, High Performance Computing) Project Goal: To apply a component-based framework to the

development of a new hybrid model for performing subsurface simulations that will combine different physical models into a single coherent simulation.

3

Different Scale ModelsDifferent Scale ModelsDifferent Scale ModelsDifferent Scale Models

Continuum Pore Sub-Pore

4

Calcite Precipitation ProblemCalcite Precipitation ProblemCalcite Precipitation ProblemCalcite Precipitation Problem

Interested in understanding (and ultimately controlling) the distribution of solid minerals that form from reaction of two dissolved chemicals (solutes).

This study will allow us to gain understanding of the impact of either high- or low-permeability inclusions along a mixing pathway on the effectiveness of mixing.

The results of modeling studies such as this will be used to design mesoscale laboratory experiments to validate our conclusions, which will in turn be used to design field-scale pilot and full-scale implementation strategies.

5

Hybrid Multiscale ModelingHybrid Multiscale ModelingBenchmark ProblemBenchmark Problem

Hybrid Multiscale ModelingHybrid Multiscale ModelingBenchmark ProblemBenchmark Problem

6

7

Project SALSSA’s Goals and Project SALSSA’s Goals and RequirementsRequirements

Project SALSSA’s Goals and Project SALSSA’s Goals and RequirementsRequirements

Create a System that … Automates and integrates research processes. Provides records for verifiability. Shares and documents: data, results, tools, and hopefully

processes. Can be used by all types of users; model developers on down to

experimentalists. Has longevity so scientists can modify the system to suit their

needs.

8

Numerical Model Configuration

Fixed1 Uncertain1

Initial Grid Generation2

Data PreparationAnd Management12

Grid Parameter Specification3

MathematicalModel Definition11

Run Numerical Model4

Continuum Workflow

Grid Refinement9

Output Visualization5

Output Analysis6

Comparative Analysis (with results of previous runs

and/or observational data)Done?

Stop

Yes

No

– R

efin

e G

rid

ParameterModification10

No

– M

odify

Par

amet

er(s

)

No

– M

odify

Mod

el

Qualitative /Quantitative

Comparisons7

HorizontalFlow

Simulation8 SummaryGraphics14

Simulation DataManagement

(I/O Documentation and Storage)13

9

Calcite Precipitation Use CaseCalcite Precipitation Use CaseCalcite Precipitation Use CaseCalcite Precipitation Use Case

Create stomp study

1. First run a single job with both the porosity and permeability the same to serve as a base case.

2. Next run a set of jobs where the fine sand (material 2) becomes progressively less permeable (decrease value by 10, 100, 1000).  keep porosity the same as case #1.

3. Starting with settings from #1, increase permeability by 10, 100,1000.  hold porosity the same.

4. Starting with settings from #1,  keep permeability the same but decrease porosity by .05 for a couple of iterations.  Again this applies to the find sand.

5. Take result where we decreased permeability by 10 and use it to create a new study.  Its not clear to me why you would start a new study.  Maybe its just an artificial case of notion of making a new study?  We could also use the case of switching to a finer grid as the cause for a new study if you think its less artificial.

Stomp Run: [Permeability (init) = Porosity (init)]

Stomp Run: [Permeability = Permeability (init) * 0.1]

Stomp Run: [Permeability = Permeability (init) * 0.01]

Stomp Run: [Permeability = Permeability (init) * 0.001]

Stomp Run: [Porosity = Porosity (init) * 0.05]

Stomp Run: [Porosity = Porosity (init) * 0.10]

Stomp Run: [Porosity = Porosity (init) * 0.15]

Stomp Run: [Permeability = Permeability (init) * 10]

Stomp Run: [Permeability = Permeability (init) * 100]

Stomp Run: [Permeability = Permeability (init) * 1000]

Stomp Study

1.

2.

3.

4.

5. Create New Stomp Study ?

10

SALSSA Components and ArchitectureSALSSA Components and ArchitectureSALSSA Components and ArchitectureSALSSA Components and Architecture

Data Services

ProvenanceStore

ContentStore

Archive

•Content management via Alfresco•User services•Pluggable metadata extraction

•Provenance in Sesame RDF store

Provenance and data managementRDF, file transfer

Organizer

User Tools Editors

•Central organizing tool•Long term interactive workflows•Data organization & access

Automated Workflow

• Job Execution• Parameter Studies• Job Monitoring• Data Archiving• Analysis “Jobs”

UpdateMessages

Provenance Recording (RDF) Analysis & Visualization

•Multiple viz tools•Techplot, Visit…

•Parallel Visualization•Hybrid visualizations•Data Analysis

Translation/ Analysis workflows

11

Applying Kepler to Subsurface Research Applying Kepler to Subsurface Research WorkflowWorkflow

Using Kepler as an End User ToolUsing Kepler as an End User Tool

Applying Kepler to Subsurface Research Applying Kepler to Subsurface Research WorkflowWorkflow

Using Kepler as an End User ToolUsing Kepler as an End User Tool

Approach End users are able to add components and tools. End users can manage their own processes using Kepler. End users would create their own workflows using pre-made higher level

actor abstractions.

Conclusion

Kepler/Virgil is NOT suitable for end users. Most of this pertains to “workflow designers” as well. Complex Types Type Checking Recording of Provenance Animation Creating Actors Managing technology Multiple Instances of Kepler Robustness

12

Applying Kepler to Subsurface Research Applying Kepler to Subsurface Research Workflow Workflow

Using Kepler for Job ExecutionUsing Kepler for Job Execution

Applying Kepler to Subsurface Research Applying Kepler to Subsurface Research Workflow Workflow

Using Kepler for Job ExecutionUsing Kepler for Job Execution

Execute Parameter Studies and Sensitivity Studies Launch and monitor multiple jobs using various queuing

systems: SGE, LSF (mpp2), fork. Monitor each job within the workflow. Notify other tools of job state. Move input/output files.

Workflow Provenance Capture Working to define an API specific for provenance

capture.

13

Issues To Address In KeplerIssues To Address In KeplerIssues To Address In KeplerIssues To Address In Kepler

Performance You can only have one instance of Kepler running on the client machine at one time. Kepler takes up a lot of memory. Possibly there could be a mechanism for packaging

just the parts you use. Kepler take a long time to startup.

Building Workflows No simple plug in model (ala spring). A mechanism to reuse/extend existing code

instead of writing new custom classes (i.e. a framework for connecting existing components instead of framework to develop components).

Better documentation for actors so that the end user does is not required to read code to understand components and know which you can hook up.

Components at too low of a level. There is a need for high level components for job launching/monitoring/file movement.

Support for parameter studies including a component for load balancing across machines.

A system built for extensibility to complex and semantic data types. A set of actors should be built for easy iteration and parameter studies. More control is needed within execution domains. (i.e. Using PN Directors inside

composite actors when a PN Director is used in the parent workflow. http://www.mail-archive.com/ptolemy-hackers@doppler.eecs.berkeley.edu/msg00381.html)

14

OrganizerOrganizerOrganizerOrganizer

15

Stomp Input WizardStomp Input WizardStomp Input WizardStomp Input Wizard

16

DemoDemoDemoDemo

Workflow Parameters numInstances: the

number of jobs that the workflow will execute.

InputData: Input data for each of the jobs.

17

Next GenerationNext GenerationNext GenerationNext Generation

SetupStomp

User works within a “Study” where a Study can be represented as a graph of processes and data inputs/outputs. Some processes are triggered by the user, others appear as by-products of user actions.

Stomp.in parameters Launch Stomp1.in Job outputsoutputsSome

Analysis graphics

Launchparameters

SomeAnalysis

Moredata

graphics

moreAnalysis

1. Baseline computation

SetupStomp

branch

2. Vary permeability in material 2 …Stomp1.in

Stomp2.in

Job

Job

outputsoutputs

outputsoutputs

Stomp2.in Job outputsoutputs

3. Vary other parameters…

18

Future SALSSA WorkFuture SALSSA WorkFuture SALSSA WorkFuture SALSSA Work

Deploy current tools to INEL to support experimental work on calcite precipitation problemApply current parameter study workflow to the CCA-based SPH code under development by Bruce PalmerIntegrate SciDAC Visualization and Analysis capabilitiesWork with SDM center to developer higher level Kepler components

Job launching, file movement, realtime monitoring

A workflow environment that combines interactive and automated workflow into one environment with appropriate user abstractions

Connect all steps into a meta workflow through provenance User control over details of view Different views of data lineage, processing steps

Extend Stomp UI wrapper to support more input optionsSupport Hybrid model and the additional processing required for setup, execution, and analysis

19

AcknowledgmentAcknowledgmentAcknowledgmentAcknowledgment

Funding for this research is provided by the U. S. Department of Energy through the following programs: Office of Science, Biological and Environmental

Research and Advanced Scientific Computing Research, Scientific Discovery through Advanced Computing (SciDAC) program.

Office of Science, Biological and Environmental Research, Environmental Remediation Sciences Program (ERSP).