NEES Grid Data Overview

Comments to Charles Severance (csev@umich.edu)

Introduction

• The data approach has evolved significantly in the past year– Second version of the Data Repository (security, access

control, performance improvements)– Data Turbine as unified real-time storage– NTCP has become increasingly capable– Model activity is bearing fruit (multiple) - Protégé / RDF /

XML Schema– Data Curation summit has provided vision– We now have a notebook which captures metadata

• As we see more detail in these areas, we find new areas that need exploration

Boxology

NEES GridData Approach

Data Models

ExperimentManagement

DataAcquisition

ExperimentMonitoring

DataAnalysis

CentralRepository

LocalRepository

Notebook

Data LifecycleData Models

Experiment Prep

Data Analysis

Data Publishing

Data Curation

Data Discovery and Reuse

Experiment Management

Data Monitoring

Data/MetadataCapture

Throughout

Data Models

Experiment Prep

Data Analysis

Data Publishing

Data Curation

Data Discovery and Reuse

Experiment Management

Data Monitoring

Data Models

• Data models are developed in RDF• Local repository supports multiple

simultaneous data models with cross-model linkages

• Metadata browser (aka Project browser) becomes the Project Browser, Notebook Browser, Site Specification Database Browser

• Metadata browser can federate multiple sources of Metadata

InstrumentationSetup

SensorGroup

Sensor

Specimen

EquipmentSetup

CalibrationSet

Equipment

CameraDataEquipment

Project Model Proj

Exp

Trial

Sensor

Site

Person Facility

Equipment

Specimen

Element Element

Site Model

Multiple Models

Notebook

Chapter Entry

Overall Data Modeling EffortsOverall Data Modeling Efforts

NEES

Site A Site CSite B

Equipment People

Experiments Trials

Equipment People

Experiments Trials

Data Data Data

TsnumaiSpecimen

Shake TableSpecimen

GeotechSpecimen

CentrifugeSpecimen

Units Sensors Descriptions

SiteSpecificationsDatabase

ProjectDescription

Domain Specificmodels

Common Elements

Data / Observations

Ref. Source: Chuck Severance

Models + Data Model

Repo

Models

Configure

Data

Load

Con

figur

e

RDF/OWL

RDF<owl:ObjectProperty rdf:ID="hasPublications"> <rdfs:domain> <owl:Class> <owl:unionOf rdf:parseType="Collection"> <owl:Class rdf:about="#Project"/> <owl:Class rdf:about="#Task"/> </owl:unionOf> </owl:Class> </rdfs:domain> <rdfs:range rdf:resource="#Publications"/> </owl:ObjectProperty>

Protégé - 2K

Models + Data Model

Repo

Models

Configure

Data

Load

Con

figur

e

RDF/OWL

RDF

Experiment Preparation

• Notebook– Allows the creation of material without needing a model– The model is pages, chapters, and “stuff”– It is all captured with data and metadata– A notebook can be attached to any object in the model

structure (i.e. a project can have a notebook, a trial can have a notebook, etc…)

• Resources• Discussions• Project Browser

– Setup basic structured metadata for the experiment - Trials, descriptions, sensors, etc… This material is captured in accordance to and with the data model

DOE ELN / Example

Setting up and Experiment

• Prior to running an experiment, the project browser will be used to create a trial, and experiment configuration, set up sensors, etc.

• In some cases, setup information may be done on the DAQ itself and the configuration information may be pulled from the DAQ

NEESgrid Experiment Data FlowNEESgrid Experiment Data Flow

NEESGridData

Repository

ProjectBrowser

DataTurbine

DataIngestion

ExperimentControl

StreamingViewer

DAQC

D

SiteSpecific

ProjectRelated

ExperimentalSetup

ExperimentalElement

DataElement

Data Model

DAQDisk

StoredViewer

Experiment Management

• Simple reference implementations for– Experiment configuration (pull / push)– Experiment Start– Experiment Stop

• Some combination of LabView and CHEF code

DT Main System

PTZ/USB

StillCapture

DT Client

BT848Video

Frames

DT Client

Capturing Video and Data

Camera ControlGateway

DAQData

CaptureDT Client

SimulationCoordinator

Site A Site B

DT Main System

Data Monitoring Tools

Still Image / Camera Control

~

< >^

^

< >

Camera ControlGateway

Creareviewers

Still imagecameracontrol

Thumb-nail

Working with Creare

• We want to leverage Creare’s live capture and viewers– Integrated Live Video and Data Viewer– Audio capability in addition to Video– JMF DataSource Capability - Use JMStudio

• SI will focus on the extraction, repository, data model, and stored viewer aspects

Data Stored in Data Turbine

Video

Stills

Data

4 5 6 7 8TimeStep*

* Time Step is only present for Pseudo-dynamic

Wall Clock Time

Data Extraction / Ingestion

• A tool will be developed to extract data from Data Turbine and place it in the NEES repository in the appropriate format– Video Channels– Image Channels– Data Channels

• The information will be stored in a format suitable for viewing using the stored viewer and appropriate metadata will be placed in the repository so that the information can be viewed

• This process is the primary new work in this plan

DT Main System

Data Extraction For Analysis

Data ExtractionPseudo-Dynamic

Continuous

Time Step Channel xyz

Start Time Step 1

End Time Step 9999

Export Auto Export

NEESData

Repository

Pseudo-Dynamic Extraction

Video

Stills

Data

4 5 6 7 8TimeStep*

Wall Clock Time

Continuous Extraction

Video

Stills

Data

Wall Clock Time

Stored Data Viewer Improvements

• Interactive Mode allowing reconfiguration of views within the Applet (insta-view)

• Linear combinations of data values• Ability to launch from the Project

Browser• Looking at integration with notebook

(i.e. launch from the notebook)

Central Repository / Curation

• Curation and the Central Repository are different than the local repository and the running / management of experiments

• Data must be packaged, kept, indexed, and maintained for the long term

Curation Flow

Project Model Proj

Exp

Trial

Sensor

Site

Person Facility

Equipment

Specimen

Element Element

Site Model

Multiple Models

Notebook

Chapter Entry

NEESgrid NEESgrid Experiment Data FlowExperiment Data Flow

NEESGridData

Repository

ProjectBrowser

DataTurbine

DataIngestion

ExperimentControl

StreamingViewer

DAQC

D

SiteSpecific

ProjectRelated

ExperimentalSetup

ExperimentalElement

DataElement

DataModel

DAQDisk

StoredViewer

DT Main System

PTZ/USB

StillCapture

DT Client

BT848Video

Frames

DT Client

Capturing Video and Data

Camera ControlGateway

DAQData

CaptureDT Client

SimulationCoordinator

Site A Site B

CurationBundle

• At some point, a project, experiment, etc is ready for curation. We must save all the information (models, notebooks, sensor data, etc) for transfer to the central repository

Data/MetadataCapture

Throughout

Data Models

Experiment Prep

Data Analysis

Data Publishing

Data Curation

Data Discovery and Reuse

Experiment Management

Data Monitoring

Workflow in Central Repository

• The workflow of the central repository will be defined over time - here are some sample concepts– Incoming materials collect in an inbox– The curator processed the materials - adds required metadata,

checks incoming data models, distinguishes information, and makes the bundle ready for publication

– Some data is published immediately, other data is held for a period of time (perhaps to allow for publication)

– Published data can be searched and viewed used and downloaded

• There are people in the curation loop• The software for this is non trivial and will evolve over

time with requirements• Sometimes it will be necessary to alter/convert data to

insure its value over time.

Workflow in Central Repository

CurationBundle

CurationBundle

InBoxProcessed

Published

Search

Hold forTime

NeedConversion

Conclusion

• This is a significant adjustment in priority• But not a significant shift in approach or

architecture• All of the elements which have been

discussed can still be delivered - the elements described herein are just higher priority.