HydroShare: An online collaborative environment for the sharing of hydrologic data and models

IN11A-1510

We envision that HydroShare will enable more rapid advances in hydrologic understanding through collaborative data sharing, analysis, and modeling. HydroShare will provide a community collaboration site that enables users to easily discover and access data and models, retrieve them to a desktop computer or perform analyses in a distributed computing environment that includes grid, cloud, or high performance computing model instances as necessary. Understanding will be advanced through the ability to integrate information from multiple sources. Outcomes (data, results, models) can then be published as new resources that can be shared with collaborators.

Our goal is to make sharing of hydrologic data and models as easy as sharing videos on YouTube or shopping on Amazon.

OCI-1148453OCI-11480902012-2017

David G. Tarboton, Ray Idaszak, Jeffery S. Horsburgh, Daniel Ames, Jonathan L. Goodall, Lawrence E. Band, Venkatesh Merwade, Alva Couch, Jennifer Arrigo, Richard P. Hooper, David W. Valentine, David R. Maidment

1

Observers and

instrumentsData

Analysis

Models

Collaboration

Publication, Archival, Curation

2

3 45

67

1. Data are observed and then loaded into CUAHSI HydroServer. 2. HydroServer stores data in a relational database and publishes it using web

services. Metadata is harvested and supports geographic and context based data discovery.

3. A HydroDesktop client user discovers, downloads and analyzes the data, or uses it in a model.

4. The user next posts the results (data and model) to HydroShare as resources (social objects) retaining provenance information on the original data source. Upon ingestion background actions (microservices) parse metadata and enable analysis based on rules and policies.

5. The user shares posted resources with colleagues (access controlled). Group collaboration and analysis on refining the analysis, model or result.

6. After iteration the result is finalized and submitted for publication.7. Upon publication the resources produced (data, model, workflow, paper) are

made immutable, access is opened and permanent persistent identifiers assigned.

Collaborative data analysis and publication use case

Collaborative integrated modeling use case

http://beta.hydroshare.org

1. Data: Links to national and global data sets of essential terrestrial variables (e.g. NASA NEX, HydroTerre)

2. Tools to preprocess and configure inputs (EcoHydroLib)3. Preconfigured models and modeling systems as services (SwatShare)4. Standards for information exchange for interoperability (OpenMI, CSDMS BMI,

DataNet Federation Consortium, DataOne)5. Tools for visualization and analysis6. Automated reasoning to couple models based on purpose, context, data and

resources

Flow

Time

Time

P

Analysis ToolsVisualization Tools

Data Loaders Data Discovery Tools

Models

Resource Repository

Resource Centric Paradigm for Modeling and Analysis

• Enable multiple models to use common “best practice” tools• Each model interacts with information in the common data store• The modeler does not need to be concerned with and can take advantage of

standardized analysis, visualization loading and discovery tools

xyt

Flow

Time

Pre-processing and model linking

Modeling Services (e.g. SWATShare)

Contact: dtarb@usu.edu Learn more: http://hydroshare.cuahsi.org

Web Browser

Drupal

iRODS

HydroDesktop

Indexing Data Grid

RESTful Interface

RESTful Interface

Policy Enforcement Point

ImplementedComing Soon

Computation

Workflow Structured Objects

GeoAnalytics Framework

Architecture and Implementation

Drupal Content Management System• Extensible Open Source Content Management Framework

for Publication written in PHP– Over 14,000 user contributed modules

• Themed and Styled Presentation of HydroShare Resources with in page visualization

• Off the shelf modules provide a Social Experience surrounding Hydrologic Data: Comments, Ratings, Group Behavior

• Custom module development supports HydroShare Data Model, GeoAnalytics and iRODS Integration

iRODS• Storage of HydroShare Resources Replicated across

multiple institutions• Rule engine for policy driven data management• Extensibility via microservices

Collaborative open source open development using GitHUB and Codeplex• http://github.com/hydroshare/• http://hydrodesktop.codeplex.com

HydroShare is currently in beta testing. The HydroShare Beta Site located at http://beta.hydroshare.org/ is open for you to use, recognizing the sometimes buggy nature of in development beta systems. So please keep backups of any content in beta.hydroshare.org that is important to you. Please provide us suggestion or feedback on problems you experience so we can improve HydroShare. Your participation as a beta tester is greatly appreciated.

Vision and Goal

Resource Data Model

HydroShare Resource

Resource Map

Aggregation

Aggregated Object1 .. *

1 .. 1 1 .. *

1 .. *

• Open Archives Initiative – Object Reuse and Exchange (OAI-ORE) - defines standards for the description and exchange of aggregations of Web resources

• BagIt – hierarchical file packaging format designed to support disk-based or network-based storage and transfer of generalized digital content

Data and metadata files

Example 1: Time Series as WaterML

• Resource Map• Science Metadata• WaterML file

Example 2: Hydrologic Model• Resource map• Science

metadata file• Model

executable file• Source folder

– Many source files

Resource Map

ScienceMetadata

source3.f90

Model.exe

sourcen.f90source1.f90 source2.f90

. . .

source

Resource Map

ScienceMetadata

WaterML

Resource TypesTime SeriesGeographic feature setOtherReferenced HIS time seriesGeographic rasterMultidimensional space time datasetRiver GeometrySample based observations (ODM2 and CZO)DocumentTabular objectsHydroDesktop project packageScriptsModelsModel componentReferenced data set from web service

CreatorContributorCoverageTitleDateDescriptionTypeFormatIdentifierLanguagePublisherRelationRightsSourceSubject

CreatedModifiedIssuedValid

NameEmailOrganizationMail addressPhone

Attributes common to all resources

SpatialTemporal

siteInfoVariableInfoQualityControlLevelMethodSourceseriesInfo

Attributes of time series (WaterML)

Child attributes…

Spatial CoverageTemporal Coverage Model TypeInstanceComponent

Attributes of Model

Begin Date&TimeEnd Date&TimeTemporal Interval

InstanceComponent

Parent ModelVersionDescriptionURL

Modeling FrameworkVersionDescriptionURL

All content is represented using the Resource Data Model that separates system and science metadata and has elements common to all resources as well as elements specific to the types of resources HydroShare will support.

Specifying a resource data type requires• Data content and logical relationships• Name and type of all data and metadata elements• Indicate whether elements are required or optional• Indicate whether elements are from a vocabulary• File formats selection or definition

Spatial ResolutionReference SystemDomain