virtual geophysics laboratory (vgl) vgl v1.2 nectar project close r.fraser, t.rankine, j.vote,...

Virtual Geophysics Laboratory (VGL)

VGL v1.2 NeCTAR Project CloseR.Fraser, T.Rankine, J.Vote, L.Wyborn, B.Evans, R.Woodcock, C.Kemp

July 2013

CSIRO | MINERALS DOWN UNDER FLAGSHIP

Hardcopy of data

SSH Client

Maths PackageProprietary Package

Let’s map it out for Geophysics …

Transform to a regular grid

Crop data to area of interest

Reformat data for processing

Upload data to HPC

resource

Configure job and start

processing

Download results

Get handed field data Visualise data

Virtual Geophysics Laboratory v1.2 | Ryan Fraser3 |

The Past• Compile raw data using proprietary FORTRAN

• Also use other proprietary software • Transform to a regular grid using more software

• MATLAB, Intrepid, ER Mapper, ESRI ArcGIS, QGIS, etc• Crop data spatially

• eg: everything in Victoria• Transform data into a file format that can be read by proprietary

scientific code• This is usually done with some handwritten python or c• There is no version control, code is often rewritten / redone

• Upload data to HPC• Manually enter input parameters/start job• Where did my results go, how do I visualise?


There seems to be a problem…

• Reproducibility – there is none• What was the input of your model?• What transformations occurred?

• It’s a manual process• Time consuming• Error prone

• Expensive • Licensing costs• Effort/time


Scientific workflow – Virtual Geophysics Laboratory (VGL)• Automated - Simplify access and integration• Reproducible process• Open access (less expense)• Allow scientists to focus on the science and not the tech• Exploiting the Cloud and HPC• Operational or education tool

• Automates and massively expands geophysics computational capacity via the Cloud


VGL

• VGL - Virtual Geophysics Laboratory• NeCTAR funded activity• Collaboration with multiple partners (CSIRO, NCI, GA, UQ, Monash, ANU)• Supporting multiple workflows• New data types and collections• New Use – Not just exploration.

Done


Hardcopy of data

SSH Client

Maths PackageProprietary Package

Let’s map it out…

Transform to a regular grid

Crop data to area of interest

Reformat data for processing

Upload data to HPC resource

Configure job and start

processing

Download results

Get handed field data Visualise data


…to thisVirtual Geophysics Laboratory

Build “science” from existing

libraries

Run jobCollect and

publish results

Discover raw data

Select spatial bounds



Introducing The Virtual Geophysics Laboratory

Data discovery

11 |

Layers discovered via remote registries

Layers consist of numerous remote data services

Virtual Geophysics Laboratory v1.2 | Ryan Fraser

Data discovery

12 |

Some data services support subsetting

Some data services support reformattinge.g. CSV, NetCDF, GeoTIFF


Data discovery

13 | Virtual Geophysics Laboratory v1.2 | Ryan Fraser

Data processing

14 |

A variety of different scientific codes are already available in the form of “Toolboxes”

Flexibility in what computing resources to utilise


Data processing

15 |

The steps so far have been building an environment to run a processing script

...or build from existing templates

Either write your own...


Managing results - provenance

Presentation title | Presenter name16 |

All of a job’s outputs are also accessible

Each job has a lifecycle that can be managed

16 |

A job’s console log can be inspected


Managing results - provenance

17 |

Successful jobs can have their entire process captured in a ISO 19115 ‘provenance record’

Each provenance record tracks all inputs, outputs, processing scripts and other metadata.... Spatial bounds...

Input/output data...


Powered by the Spatial Information Services StackCommon Platform

MarineEnvironment, Water

Groundwater GeologyGeophysics


VGL: Opportunities

• Free for research use• Supporting multiple workflows• Model Registry (3D) – researcher contributed models• New Scientific Codes – Underworld, eScript, UBC, Airborne EM inversion

codes + others (relatively “easy” to add)• Data sets from GA: National Airborne Geophysical DB including

– Gravity, Radiometric, AEM, Magnetics– Data sets from State Surveys

• Broad application and generic implementation• Scalable Infrastructure (Cloud)


What’s next?

• VGL sustainability strategy kick-ins • More Data and More Codes

• Extensions for new domains beyond the geosciences• Reuse of infrastructure by additional clients• Commercial connections• Roadshow for state government geologic surveys (funded by

AuScope)• Provenance Service – take learnings of VGL and research to create

a generic prov service

20 | Virtual Geophysics Laboratory v1.2 | Ryan Fraser

Thank you

Ryan FraserResearch Manager

t +61 8 6436 8760e [email protected] siss.auscope.org

VGL is Power by:

VGL acknowledges funding from the NeCTAR project http://www.nectar.org.au NeCTAR is an Australian Government project conducted as part of the Super Science initiative and financed by the Education Investment Fund.

VGL is a collaboration between:

VGL is supported by: University of Queensland, Monash University and the Australian National University

http://www.nectar.org.au/

virtual geophysics laboratory (vgl) vgl v1.2 nectar project close r.fraser, t.rankine, j.vote,...

Documents

data discovery

ryan fraser slide

processing upload data

handed field data

redone upload data

regular grid crop data

past compile raw data

flagship slide