U.S. Department of the InteriorU.S. Geological Survey

Data Integration Progress and Guiding Principles

Disciplines, generalization, and open-access.

David Blodgett – dblodgett@usgs.govUSGS Office of Water Information Center for Integrated Data Analytics

Outline

· Data Integration Disambiguation

· Barriers to moving Forward.

· Anecdotes, everyone loves anecdotes!

· Principles to go Forward!

Disclosures

· I’m a water guy.

· I‘m a millennial.

· I assume Internet.

· I’m a Badger.· … Forward!

Data Integration – Disambiguated.

Integration is the act of combining multiple things into a whole.

Data Integration – Disambiguated.

What makes something integrated?How different do things need to be to count?Do you just need to combine things?

What kind of data integration is needed for decisions?

2014-

05-

12

Jeff.deLaBeaujardiere@noaa.gov

7

Visual Integration

Data Consolidation

DataWarehouse

Data Bundling Data Fusion

Integrated Search Multi-source Data Ingest

in the Cloud?

Application / Decision Driven Model of Data Integration Slide Credit: Jeff de La Beaujardiere

What kind of data integration is needed for decisions?

2014-

05-

12

Jeff.deLaBeaujardiere@noaa.gov

8

Visual Integration

Data Consolidation

DataWarehouse

Data Bundling Data Fusion

Integrated Search Multi-source Data Ingest

in the Cloud?

A general model for data integration.

Disciplinary Details

Free and Open Service Access

Generalized Standards

Service OrientationOn local machines, we run software.

List, introspect, summarize, transform, integrate.Can scan the entire domain of the data!

A service may do any or all of these things.

Software on the server can summarize the domain and range of its holdings. (ie. Deliver Dynamic Metadata)

Web Service – So what?

Software on the server

can summarize the

domain and range of its

holdings.

Generalized Aspects of Data Services

Spatial/ Temporal

Extent

Attribute Extent

Blob of Bits

Available Formats

International Standards.

Various Communities’ Interchange

Discipline specific linked to other disciplines.

Practical Barriers

‘I don’t know how to use the required software.’

‘The software I need is really expensive.’

‘The information I need is a big mess.’

‘The information I need is really big.’

Understanding Barriers

‘The information is in a language I don’t know.’

‘The information is in a format I’ve never seen.’

‘The taxonomy used doesn’t work with mine.’

‘I’m not sure if what I’m seeing is a data quality issue or real.’

Defensive Barriers

‘I collected this data and want to publish on it.’

‘People won’t interpret my data correctly.’

‘I don’t want to be liable for decisions made.’

‘This data’s quality is too low to stand behind.’

Square Pegs and Round Holes

Coverages and Features

A grid cell IS NOT a point measurement!!!

Scale Discontinuity

Anecdotes!...

Because they are instructive!

Water Quality Portal

http://www.waterqualitydata.us

USGS, EPA, USDA Joint service providing water quality and other environmental monitoring data.

Integrated Ocean Observing System

Weather Underground 42K Current Conditions Weather Stations!

WeatherCommon architecture for access and processing multiple environmental data resources!

Geo Data Portal Data Integration Framework

Center for Integrated Data Analytics: Nate Booth, Tom Kunicki, Dave Blodgett, Jordan Walker, Ivan Suftin, I-Lin Kuo.

Landscape

Climate

Enabling Technologies….

____.data.gov – Big Win!

Data access type is a first class citizen!

Includes both human and machine metadata.

Machine-interpretability is an expectation.

Content management systems and catalogs are becoming data service providers!!!

Forward!

Principle #1: Data Object Patterns

We must continue to identify and model the common patterns our data adhere to.

Non-interpretive content / attributes should be provided by service ‘methods’.

These patterns must transcend discipline or implementation.

Principle #2: Domain Semantics.

Semantic relationships are necessarily governed by a given scientific domain itself.

This is Foundational to all additional interdisciplinary concerns.

Principle #3: __ - Agnostic Standards

Standards, specifications, and best practices must be ____ - agnostic.

A standard can be implemented using any technology, in any discipline.

eg. WaterML2 -> TimeSeriesML

Principle #4: Identity Management

Uniqueness can’t be taken for granted and must be curated very deliberately.

You are not your location. Neither is a place.

Foundational to linking any and all information to an entity.

A few thoughts to leave you with…

Maps are metadata.

Index-based data access is dead.

A Geospatial database should be coherent without it’s spatial table.

Summary

A standard is an established generalization.

Scientific disciplines govern their semantics.

Open-access (the internet) must be a given.