cuahsi hydrologic information system: web services and related technologies
DESCRIPTION
CUAHSI Hydrologic Information System: web services and related technologies. WaterML & Web Services team: David Valentine Tom Whitenack Tim Whiteaker Matt Rodriguez. HIS PIs: David Maidment Ilya Zaslavsky David Tarboton Michael Piasecki Jon Goodall - PowerPoint PPT PresentationTRANSCRIPT
CUAHSI Hydrologic Information
System:web services and related
technologiesHIS PIs: David Maidment Ilya Zaslavsky David Tarboton Michael Piasecki Jon Goodalland Rick Hooper (CUAHSI)
WaterML & Web Services team: David Valentine Tom Whitenack Tim Whiteaker Matt Rodriguez
www.cuahsi.org/his/
Test bed HISServers
Central HIS servers
ArcGIS
Matlab
IDL
MapWindow
Excel
Programming (Fortran, C, VB)
Desktop clients
Customizable web interface (DASH)
HTML - XMLW
SDL - SO
AP
Hydrologic Information System Service Oriented Architecture
Modeling (OpenMI)
Global search (Hydroseek)
WaterOneFlow Web Services, WaterML
HIS LiteServers
External data providers
Deployment to test beds
Other popular online clients
ODM DataLoader
Streaming Data Loading
Ontology tagging (Hydrotagger)
WSDL and ODM registration
Data publishing
ODMTools
Server config tools
CUAHSI Member Institutions
112 Universities as of June 2007
HIS Goals
• Hydrologic Data Access System – better access to a large volume of high quality hydrologic data
• Support for Observatories – synthesizing hydrologic data for a region
• Advancement of Hydrologic Science – data modeling and advanced analysis
• Hydrologic Education – better data in the classroom, basin-focused teaching
Project co-PI in Phase 2
Collaborator in Phase I
CUAHSI HIS Partner Institutions
10 WATERS Testbed Sites
Windows 2003 Server4 GB Ram500 GB DiskQuad Core CPU
SQLServer 2005IIS (Internet Information Server)
DASH ArcGIS Server
Visu
al S
tudi
o 20
05
WaterOneFlow Web ServicesArcGIS 9.2
GIS Data Mxd Services
ODDa
taLo
ader
ODM
tool
s
COTS Software
HIS Applications Data
NWIS DV
NWIS IID
Your ODM
HISTeam
WATERSTestbed
WATERS Network Information System
SupercomputerCenters:NCSA,TACC
Domain Sciences:
Unidata, NCARLTER, GEON
Government:USGS, EPA,
NCDC, USDA
Industry:ESRI, Kisters,
OpenMI
HISTeam
WATERS Testbed
WATERS Network Information System
CUAHSI HIS
International Partners
CSIRO Land and Water ResourcesWater Resources Observations Network
(WRON)
European CommissionWater database design and model integration
(HarmonIT and OpenMI)
Digital WatershedHow can hydrologists integrate observed and
modeled data from various sources into a single description of the environment?
Rainfall & SnowWater quantity
and quality
Remote sensing
Water Data
Modeling Meteorology
Soil water
Water Data Web Sites
NWISWeb site output# agency_cd Agency Code# site_no USGS station number# dv_dt date of daily mean streamflow# dv_va daily mean streamflow value, in cubic-feet per-second# dv_cd daily mean streamflow value qualification code## Sites in this file include:# USGS 02087500 NEUSE RIVER NEAR CLAYTON, NC#agency_cd site_no dv_dt dv_va dv_cdUSGS 02087500 2003-09-01 1190USGS 02087500 2003-09-02 649USGS 02087500 2003-09-03 525USGS 02087500 2003-09-04 486USGS 02087500 2003-09-05 733USGS 02087500 2003-09-06 585USGS 02087500 2003-09-07 485USGS 02087500 2003-09-08 463USGS 02087500 2003-09-09 673USGS 02087500 2003-09-10 517USGS 02087500 2003-09-11 454
Time series of streamflow at a gaging station
USGS has committedto supporting CUAHSI’sGetValues function
Observation Stations
Ameriflux Towers (NASA & DOE) NOAA Automated Surface Observing System
USGS National Water Information System NOAA Climate Reference Network
Map for the US
Water Quality Measurement Sites in EPA Storet
Substantial variation in data availability from states
Data from Bora Beran, Drexel University
Water Quality Measurement Sites from Texas Commission for Environmental Quality (TCEQ)
Geographic Integration of Storet and TCEQ Data in HIS
Point Observations Information Model
• A data source operates an observation network• A network is a set of observation sites• A site is a point location where one or more variables are measured• A variable is a property describing the flow or quality of water• An observation series is an array of observations at a given site, for a given variable, with start time and end time• A value is an observation of a variable at a particular time• A qualifier is a symbol that provides additional information about the value
Data Source
Network
Sites
ObservationSeries
Values{Value, Time, Qualifier}
USGS
Streamflow gages
Neuse River near Clayton, NC
Discharge, stage, start, end (Daily or instantaneous)
206 cfs, 13 August 2006
Return network information, and variable information within the network
Return site information, including a series catalog of variables measured at a site with their periods of record
Return time series of values
Observations CatalogSpecifies what variables are measured at each site, over what time interval,
and how many observations of each variable are available
Observations Data Model Schema (version 4.0)
Data Source and Network Sites Variables Values Metadata
Depth of snow pack
Streamflow
Landuse, Vegetation
Windspeed, Precipitation
Controlled Vocabulary Tables
e.g. mg/kg, cfs
e.g. depth
e.g. Non-detect,Estimated,
A site is a point location where one or more variables are measured
A data source operates an observation network A network is a set of observation sites
Metadata provide information about the context of the observation.A variable is a property describing the flow or quality of water
A value is an observation of a variable at a particular time
From Ernest To, David Maidment, CRWR
WaterOneFlow• Set of query functions • Returns data in WaterML
NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, USGS SNOTEL, ODM (multiple sites)
WaterML key elements
• Response Types
– SiteInfo
– Variables
– TimeSeries
• Key Elements– site– sourceInfo– seriesCatalog– variable– timeSeries
• values
– queryInfo
GetValues
GetVariableInfo
GetSiteInfo
variables
variablesResponse
variable
1
many
timeSeriesqueryInfo
criteria
timeSeriesResponse
variable
sourceInfo
queryURL
values
site queryInfo
criteria
sitesResponse
seriesCatalog siteInfo
queryURL
variable
series
variableTimeInterval
1
many
Structure of responses
More Information about WaterML…
next 20 slides…/we may skip them/
Or check the specification online at http://www.opengeospatial.org/standards/dp
DatasetInfoTypeSiteInfoType
Elements Defining Spatial Location
for observation sites for continuous surfaces
(other site information)
SourceInfoType
GeogLocationType
(other dataset information)
LatLonPointType
childelements
GeogLocationType
LatLonBoxType
LatLonPointType
SiteInfoResponseType
• Namespaces• queryInfo• site
Network
SitesVariables
queryInfo example
userparameters
query URL
• Parameters sent to service• URLs called (if external resource)
siteInfo
• Name• Site Code• Location
geoLocation
• geogLocation – geographic coordinates– LatLon point– LatLon box
• localSiteXY – projected coordinates
series
• variable – what is measured• valueCount – how many measurements• variableTimeInterval – when is it
measured
TimePeriodType
variable
• variableCode – global identifier• variableName• units
Sites
VariablesValues
TimePeriodType
variableTimeInterval• TimePeriodType – date range (including
“last n days”• TimeInstantType – single measurement
Compare with…
SiteInfo responsequeryInfo
site
name
code
location
seriesCatalog
variables
Series how many
when
TimePeriodType
VariablesResponseType
• variable – same as in series element• Code, name, units Sites
VariablesValues
TimeSeriesResponseType
• queryInfo• timeSeries
– sourceInfo – “where”– variable – “what”– values
Sites
Variables
Values
sourceInfo• SiteInfoType
– Same as siteInfo element– code, name, location
• DataSetInfoType– For data continuous in space– LatLonPointType– LatLonBoxType
values
• Each time series value recorded in value element
• Timestamp, plus metadata for the value, recorded in element’s attributes
ISO Time
valuequalifier
value metadata examples
• qualifiers• censorCode (lt, gt, nc)• qualityControlLevel (Raw, QC’d, etc.)• methodID• offset
– offsetValue– offsetUnitsAbbreviation– offsetDescription– offsetUnitsCode
TimeSeries responsequeryInfo
location
variable
values
OGC Harmonization Best Practices
• WaterML text includes steps for harmonizing with GML/O&M
• Align spatial feature descriptions (e.g. using gml:Point, gml:Envelope)
• Align service signatures (getCapabilities)• Align terminology with O&M
More on OGC Interactions• WaterML is published as OGC Discussion paper• We are working with OGC O&M (Observations &
Measurements) authors to reconcile WaterML with OGC specs:– WOML proposal (Water Observations ML)
• Think of it as WFS + O&M type service (with GetSiteInfo proxied as OGC’s GetFeatureInfo request)
– The plan is to provide WOML-compliant services alongside WaterML 1.1
– There is an international group that Dave Valentine is coordinating, focused on developing a standard water information exchange schema (started at WaterML workshop in Canberra in September’07)
• As we are going with OGC through this harmonization, we shall be able to assist WQX with similar mappings (more below on mapping between WaterML and WQX)
Hydrologic Information Server
Microsoft SQLServer Relational Database
Observations Data & Catalogs Geospatial Data
GetSites
GetSiteInfo
GetVariables
GetVariableInfo
GetValues
DASH – data access system for hydrologyWaterOneFlow services
ArcGIS Server
SQL Server
ODMs and catalogs. All instancesexposed as ODM (i.e. have standard ODM tables or views: Sites, Variables, SeriesCatalog, etc.)
NWIS-IID
NWIS-DV
ASOS
STORET
TCEQ
BearRiver. . .
Spatial store
Geodatabase or collection of shapefilesor both
NWIS-IID points
NWIS-DV points
ASOS points
STORET points
TCEQ points
BearRiver points. . .
My new ODM
My new pointsMore databases
More synced layers
DASH Web Application
Background layers
(can be in the same or separate spatial store)
WOF services
Web services from a common template
NWIS-IID WS
NWIS-DV WS
ASOS WS
STORET WS
TCEQ WS
BearRiver WS. . .
My new WS
More WS fromODM-WS template
USGS
NCDC
EPA
TCEQ
Web Configuration fileStores information about registered networks
MXDStores information about layers
WSDLs
, web
servi
ce U
RLs Connection
strings
Layer info,
symbology, etc.
ODMDataLoader
2
6
5
3
1
4
WORKGROUPHISSERVERORGANIZATION
STEPS FORREGISTERINGOBSERVATIO
NDATA
New network registration steps
Using the ODM DataLoader or other, load your data into a blank ODM instance (this will create all ODM tables that HIS relies on)
Copy Web Services template to a new folder, edit the template web.config file to point to the new ODM, test to make sure the new service works as expected
Create a point layer (a feature class in GDB, or a shapefile) from the new ODM’s Sites table using the GetSitesTool.
Add the point layer to the MXD document, specify symbology, scale-dependent rendering, etc.
Add information about the new ODM, the associated web service, and the associated point layer, to HIS configuration file (see the first slide for the exact content)
Restart the HIS serviceRegister and test the new service at the HIS Central:
http://water.sdsc.edu/centralhis/6
5
4
3
2
1
7
Central HIS Data
Services
Catalog
Beyond Syntactic Uniformity: Semantic MediatorWhat we are doing now …..
NWIS
NAWQA
NARR
generic request
GetValues
GetValues
GetValues
GetValues
GetValues
GetValuesGetValues
GetValues
GetValues ODM
Michael PiaseckiDrexel University
Hydroseekhttp://www.hydroseek.net
Supports search by location and type of data across multiple observation networks including NWIS, Storet, and university data
Semantic Tagging of Harvested Variables
CUAHSI HIS as a mediator across multiple agency and PI data
– Keeps identifiers for sites, variables, etc. across observation networks
– Manages and publishes controlled vocabularies, and provides vocabulary/ontology management and update tools
– Provides common structural definitions for data interchange
– Provides a sample protocol implementation– Governance framework: a consortium of
universities, MOUs with federal agencies, collaboration with key commercial partners, led by renowned hydrologists, and NSF support for core development and test beds
US Map of USGS Observations
Antarctica
Puerto Rico
Hawaii
Alaska
Different types of nutrients by decade: Available Data Total
Some physical properties by decade: Available Data Total
Measurement Counts by Decade
Measurement counts by Decade in the Western States
STORET DatacubeThe EPA STORET datacube contains 273K sites and 2.7M series. The EPA STORET datacube contains 273K sites and 2.7M series.
Florida is the source Florida is the source of about 25% of the of about 25% of the total records. total records.
Number of years of record by start decadeNumber of years of record by start decade
About 60% of the About 60% of the water quality records water quality records are short term are short term measurements (one measurements (one year or less in year or less in duration). The starting duration). The starting decade of the longer decade of the longer series are to the right.series are to the right.
93% of the series are93% of the series arewater quality data. water quality data.
Demo