ncai exchange network tribal users meeting april 9, 2008 tribal water quality data management
TRANSCRIPT
NCAI Exchange Network NCAI Exchange Network
Tribal Users MeetingTribal Users Meeting
April 9, 2008April 9, 2008
Tribal Water Tribal Water Quality Data Quality Data ManagementManagement
OverviewOverview
Tribal 106 Grants Requirements for Tribal 106 Grants Requirements for Monitoring and AssessmentMonitoring and Assessment
Essential concepts of Water Quality Data Essential concepts of Water Quality Data Management Management
Understanding WQXUnderstanding WQX Steps to a WQX SubmittalSteps to a WQX Submittal Accessing DataAccessing Data Upcoming DevelopmentsUpcoming Developments How to decide where to beginHow to decide where to begin
Tribal 106 Guidance Tribal 106 Guidance RequirementRequirement
All tribes must submit a TAR (Tribal All tribes must submit a TAR (Tribal Assessment Report) to continue Assessment Report) to continue eligibility for 106 funds (8-3)eligibility for 106 funds (8-3) Monitoring Strategy Monitoring Strategy Water Quality AssessmentWater Quality Assessment STORET-compatible electronic dataSTORET-compatible electronic data
106 Guidance parameters
Fundamental Programs:- Dissolved Oxygen (DO)- Temperature- pH- Turbidity
Intermediate Programs:- Nitrogen - Phosphorus
Mature Programs:- Pathogens- Macroinvertebrates- Physical Habitat indicators
I. Monitoring StrategyII. Monitoring Objectives III. Monitoring DesignIV. Core IndicatorsV. Quality AssuranceVI. Data Management VII. Data Analysis/Assessment VIII. Reporting IX. Programmatic EvaluationX. General Support and Infrastructure
10 Elements of a Tribal Monitoring Strategy
Essentials of Data ManagementEssentials of Data Management
Monitoring data consists of:Monitoring data consists of: Where sample/measurement was collectedWhere sample/measurement was collected When it was collectedWhen it was collected What was collectedWhat was collected How it was collectedHow it was collected Why it was collectedWhy it was collected Who collected itWho collected it
It does not take a lot of information to create a It does not take a lot of information to create a good data set.good data set.
How does data management fit in?How does data management fit in?
The life of a sample begins here
Labs generate data
Field Sheets capture data
Now what to do with all this data?
First, make sense of what you haveFirst, make sense of what you have
Organize data electronically
Use the tools you’re familiar with:
•Access
•Excel
•Other higher-powered database management system (i.e. Oracle)
Have a plan for growth
Not only will your skill grow, but your data volume will grow. Be prepared for it.
Find out what’s already available
Data are valuable, plan for re-useData are valuable, plan for re-use
Electronic data are more valuable than data in file cabinets
The more data are re-used, the more valuable they become
Shared data are of even higher value
•Provide for better planning decisions
•Incentivize collaborative efforts
•Make the most use of the monitoring $$s being invested
How EPA Describes Monitoring DataHow EPA Describes Monitoring Data
ProjectsProjects (Why data were collected) (Why data were collected)
- Brief summary of monitoring plan- Brief summary of monitoring plan Monitoring LocationsMonitoring Locations (Where data were (Where data were
collected)collected)
- Describe where monitoring takes place- Describe where monitoring takes place ResultsResults (When, How and What were (When, How and What were
collected)collected)
- Measurements of what were monitored- Measurements of what were monitored
Three categories of data:
WQX is designed around these WQX is designed around these conceptsconcepts
ORGANIZATION
ORG Description
ORG Electronic Address
ORG Phone
ORG Address
MONITORING LOCATION
Monitoring Location Identity
Monitoring Location Geospatial
Monitoring Location Binary Object
RESULT
Result Description
Result Binary Object
Result Analytical Method
Result Lab Info
Result Detection Quantitation Level
Lab Sample Prep
MONITORING ACTIVITY
Activity Description
Activity Location
Sample Description
Sample Prep
Subsample Description
Activity Binary Object
PROJECT
Project Description
Project Binary Object
ACTIVITY GROUP
Activity Group Name
Activity Group Type
Activity IDs
The WQX SchemaThe WQX Schema The WQX schema is a definition of the The WQX schema is a definition of the
structure and format that data must have structure and format that data must have to be submitted through WQX. to be submitted through WQX.
Primary components: Organization, Primary components: Organization, Project, Monitoring Location, Monitoring Project, Monitoring Location, Monitoring Activity, Samples, and ResultsActivity, Samples, and Results
Data elements currently accommodate Data elements currently accommodate physical, chemical, and fish tissue dataphysical, chemical, and fish tissue data
Biological monitoring and habitat data Biological monitoring and habitat data flow will be incorporated in the very near flow will be incorporated in the very near futurefuture
Submitting data using Submitting data using WQXWQX
Water Quality Exchange (WQX) uses Water Quality Exchange (WQX) uses the Exchange Network to flow the Exchange Network to flow physical, chemical and fish tissue, physical, chemical and fish tissue, and soon biological and habitat data and soon biological and habitat data from an tribe’s own database to the from an tribe’s own database to the existing STORET Warehouse existing STORET Warehouse
Tribes that have been awarded Tribes that have been awarded Exchange Network Grants will be Exchange Network Grants will be able to submit data through WQXable to submit data through WQX
CDXWQX
Parse and Load Software
WQXODS
Data Submitter
XML Submission
Schema ValidationReceive Error Responses
Approved XMLSubmission
Data Context Error Report
Data Load Confirmation and Summary
How WQX WorksHow WQX Works
WQX in the big pictureWQX in the big picture
Steps to Submitting to Steps to Submitting to WQXWQX
1.1. Review the resources that are availableReview the resources that are available2.2. Getting involved with the Exchange Getting involved with the Exchange
NetworkNetwork3.3. Getting a NAAS AccountGetting a NAAS Account4.4. Getting an Organization IDGetting an Organization ID5.5. Understanding XML (How does my data Understanding XML (How does my data
relate to an XML file?)relate to an XML file?)6.6. Submitting a fileSubmitting a file7.7. Understanding errorsUnderstanding errors8.8. Understanding the WQX structureUnderstanding the WQX structure9.9. Interacting with WQXInteracting with WQX
Step 1. Resources Step 1. Resources AvailableAvailable
The STORET/WQX team has several resources available to help The STORET/WQX team has several resources available to help you:you:
Available at the STORET web site: www.epa.gov/storetAvailable at the STORET web site: www.epa.gov/storet Visit the WQX section: http://www.epa.gov/storet/wqx.htmlVisit the WQX section: http://www.epa.gov/storet/wqx.html Diagrams and presentations about how WQX worksDiagrams and presentations about how WQX works Presentations from others on how they’ve implemented Presentations from others on how they’ve implemented
WQX (including tribes)WQX (including tribes) Lists of allowable values for WQXLists of allowable values for WQX
Available at the Exchange Network Site: Available at the Exchange Network Site: http://www.exchangenetwork.net/exchanges/water/wqx.htmhttp://www.exchangenetwork.net/exchanges/water/wqx.htm
Read the Flow Configuration DocumentRead the Flow Configuration Document See example XML FilesSee example XML Files Examine the Data Exchange TemplateExamine the Data Exchange Template
Last but not least, don’t be afraid to ask for help:Last but not least, don’t be afraid to ask for help: Join the STORET list serve: Join the STORET list serve: www.epa.gov/storet/listserv.htmlwww.epa.gov/storet/listserv.html Call the STORET hot line: 1-800-424-9067Call the STORET hot line: 1-800-424-9067 end an email to [email protected] an email to [email protected]
Step 1. Remember thisStep 1. Remember this
All of this information is available at: All of this information is available at: www.epa.gov/storetwww.epa.gov/storet
Step 2. Getting involved Step 2. Getting involved in the Exchange Networkin the Exchange Network Ask yourself the following questions:Ask yourself the following questions:
1.1. Does my organization have a node?Does my organization have a node?2.2. Have we submitted data for another Have we submitted data for another
flow?flow?3.3. Did we use a node? Node client? Or Did we use a node? Node client? Or
CDX Web?CDX Web?4.4. Who is my Node Administrator? Do we Who is my Node Administrator? Do we
have one?have one? If the answer to these questions is If the answer to these questions is
‘No’ or ‘I don’t know’ visit: ‘No’ or ‘I don’t know’ visit: www.exchangenetwork.netwww.exchangenetwork.net
Step 3. Getting a NAAS Step 3. Getting a NAAS AccountAccount
The NAAS or Network The NAAS or Network Authentication Authorization Service Authentication Authorization Service is a user account that you can use to is a user account that you can use to access the Exchange Network. They access the Exchange Network. They can be specific for an individual can be specific for an individual node or for a particular usernode or for a particular user
NAAS accounts are assigned by NAAS accounts are assigned by Node Administrators. If you don’t Node Administrators. If you don’t have a Node Administrator, contact have a Node Administrator, contact the CDX help desk at: 888-890-1995 the CDX help desk at: 888-890-1995 Or [email protected] [email protected]
Step 4. Getting an Org Step 4. Getting an Org IDID
An Organization ID (Org ID) is a unique An Organization ID (Org ID) is a unique identifier assigned to your organization identifier assigned to your organization (i.e. YUROK for the Yurok Tribe).(i.e. YUROK for the Yurok Tribe).
Org IDs are assigned by EPA upon the Org IDs are assigned by EPA upon the request of the user.request of the user.
You must have an Org ID to be able to You must have an Org ID to be able to submit to WQXsubmit to WQX
To request an Org ID, send an email to To request an Org ID, send an email to [email protected]@epa.gov
Provide the NAAS accounts that you want Provide the NAAS accounts that you want to have access to your Organizationto have access to your Organization
Step 5. Understanding Step 5. Understanding XMLXML
XML can be intimidating at firstXML can be intimidating at first There are many ways to create XML, some things to There are many ways to create XML, some things to
do:do:1.1. The sample files and Data Exchange Template are The sample files and Data Exchange Template are
key to understanding how to create a WQX XML file.key to understanding how to create a WQX XML file.2.2. Find a programmer who can spend 20-40 hours on Find a programmer who can spend 20-40 hours on
designing an XML filedesigning an XML file3.3. Have some test data that are representative of the Have some test data that are representative of the
data you’ll be submittingdata you’ll be submitting4.4. Test the procedures that the programmer developed Test the procedures that the programmer developed
(plan for an additional 20-40 hrs testing)(plan for an additional 20-40 hrs testing)5.5. Start small, it takes time to create an XML file, don’t Start small, it takes time to create an XML file, don’t
waste development time waiting for a computer to waste development time waiting for a computer to process a file.process a file.
***You should only have to do this once, after that, you’re ***You should only have to do this once, after that, you’re just running the procedure.just running the procedure.
Step 6. Submitting a FileStep 6. Submitting a File Once you have a NAAS account and an Org Once you have a NAAS account and an Org
ID, and have successfully created an XML file, ID, and have successfully created an XML file, you’re ready to submit.you’re ready to submit.
The original test file should be small, but The original test file should be small, but representative of the data you’ll be submittingrepresentative of the data you’ll be submitting
Submit to CDX test firstSubmit to CDX test first Once you get a successful submission, keep Once you get a successful submission, keep
track of the transaction IDtrack of the transaction ID Send an email to the CDX help desk:…. Send an email to the CDX help desk:….
Requesting access to WQX production. Requesting access to WQX production. Provide them the transaction ID of the Provide them the transaction ID of the successful submission.successful submission.
Go ahead an begin submitting to productionGo ahead an begin submitting to production
Step 7. Understanding Step 7. Understanding ErrorsErrors
There are two types of errors in WQX:There are two types of errors in WQX:
1.1. Schema ErrorsSchema Errors: the submitted file does : the submitted file does not conform to the WQX Schema. not conform to the WQX Schema. (Report is available in the file ‘Validation (Report is available in the file ‘Validation Results’ which is an xml file).Results’ which is an xml file).
2.2. Processing ErrorsProcessing Errors: the submitted file : the submitted file does not pass all of the WQX Business does not pass all of the WQX Business Rules (Report is available in the file Rules (Report is available in the file ‘ProcessingReport.zip’)‘ProcessingReport.zip’)
Step 8. Understanding Step 8. Understanding the WQX Structurethe WQX Structure
WQX files should be organized as WQX files should be organized as follows:follows:
Header (with processing instructions)Header (with processing instructions) OrganizationOrganization
Organizational InformationOrganizational Information ProjectsProjects Monitoring LocationsMonitoring Locations ActivitiesActivities
Linkage to ProjectsLinkage to Projects Linkage to Monitoring LocationsLinkage to Monitoring Locations ResultsResults
Activity GroupsActivity Groups
Step 9. Interacting with Step 9. Interacting with WQXWQX
WQX is designed to function in a WQX is designed to function in a Node-to-Node environment (i.e. you Node-to-Node environment (i.e. you manage your data, let your manage your data, let your computer and Node keep EPA in computer and Node keep EPA in sync).sync).
WQX allows Inserts, Updates, and WQX allows Inserts, Updates, and Deletes via a batch processDeletes via a batch process
WQX has Outbound services via the WQX has Outbound services via the Query and Solicit method data Query and Solicit method data visualization and system visualization and system synchronization purposes.synchronization purposes.
Accessing DataAccessing DataThe National STORET Data Warehouse serves as the national repository for Modern STORET data as well as WQX data. Users may obtain data from the warehouse through the STORET home page : www.epa.gov/storet
Web services providing direct access to warehouse data will also provide enhanced accessibility to the data.
STORET Warehouse STORET Warehouse UpdatesUpdates
The National STORET Data The National STORET Data Warehouse enhancements for Warehouse enhancements for improved data accessibilityimproved data accessibility New Watershed Summary – Available New Watershed Summary – Available
NowNow New Web Services – Available NowNew Web Services – Available Now
Please visit the STORET website for Please visit the STORET website for more information : more information : www.epa.gov/storetwww.epa.gov/storet
Where do Web Services Where do Web Services fit?fit?
Data in/out services via WQX
Data out services through direct access to the WH
WH services available via Exch. Net.
All Done with Web Services!!
An example using An example using Monitoring DataMonitoring Data
What’s coming next?What’s coming next? WQX 2.0 is near completionWQX 2.0 is near completion
Adds Biological data and Habitat data to the Adds Biological data and Habitat data to the schemaschema
Supports Bio/Habitat metrics and indexesSupports Bio/Habitat metrics and indexes EPA will continue to support WQX 1.0 for the EPA will continue to support WQX 1.0 for the
foreseeable futureforeseeable future XML Generation ToolXML Generation Tool
Tool that will support organizations that don’t Tool that will support organizations that don’t have the ability to create XMLhave the ability to create XML
Batch Geo-location toolBatch Geo-location tool Tie Monitoring Locations to the National Tie Monitoring Locations to the National
Hydrography DatasetHydrography Dataset
Dates to RememberDates to Remember WQX Version 1.0 data flow in production nowWQX Version 1.0 data flow in production now
Physical, chemical, microbiological, and fish tissue Physical, chemical, microbiological, and fish tissue data onlydata only
Add Biological/Habitat data to WQX schemaAdd Biological/Habitat data to WQX schema Bio/Habitat WQX in production (May 08)Bio/Habitat WQX in production (May 08)
Develop Web tool for XML GenerationDevelop Web tool for XML Generation Tool available for general use (September 08)Tool available for general use (September 08)
Batch Geo-location ToolBatch Geo-location Tool Under development available January 2009Under development available January 2009
What is the XML What is the XML Generation Tool?Generation Tool?
A web-based application that assists A web-based application that assists a data provider with the creation of a data provider with the creation of a valid submission file for the Water a valid submission file for the Water Quality Exchange.Quality Exchange.
The next generation of SIM and The next generation of SIM and WebSIM, redesigned to target WQX.WebSIM, redesigned to target WQX.
This tool is meant to allow those who do not have This tool is meant to allow those who do not have Exchange Network Nodes the ability to share Exchange Network Nodes the ability to share data over the webdata over the web
The tool will be easy to use and accept basic The tool will be easy to use and accept basic input formats (i.e. Excel Spreadsheets).input formats (i.e. Excel Spreadsheets).
Data submitted via this tool will be available on Data submitted via this tool will be available on the internet within one week of submission.the internet within one week of submission.
Data-sharing for allData-sharing for all
WQX WQX DatabaseDatabase
CDX WebCDX Web• Online Registration• Log In
How the XML Generation Tool How the XML Generation Tool fits infits in
CDX CDX NodeNode
XML Generation XML Generation ToolTool• Import Flat File• Apply Defaults & Translations• Validate & Correct• Export XML
State/Tribal State/Tribal NodeNode• Submit• Get Status• Download
CDX CDX LiteLite• Submit• Get Status• Download
State/Tribal State/Tribal DatabaseDatabase
State/Tribal State/Tribal SystemSystem• Retrieve Data from Database• Export XML
WQX SystemWQX System• Load XML File• Update WQX Database• Create Processing Report
DemoDemo
By the Way, what happened to National WebSIM?
Batch Geo-LocatingBatch Geo-Locating
New development that EPA is working onNew development that EPA is working on Allow for any new stations submitted via WQX Allow for any new stations submitted via WQX
to be automatically georeferenced to the to be automatically georeferenced to the NHD+NHD+
The Georeferenced reach codes would be The Georeferenced reach codes would be returned to the data provider, as well an returned to the data provider, as well an opportunity to QA the georeferenced locationsopportunity to QA the georeferenced locations
This will use a concept called ‘Rain Drop’ to This will use a concept called ‘Rain Drop’ to ensure that points are tied to the correct NHD ensure that points are tied to the correct NHD reachreach
• If the two points represent Accuracy/Precison issues then note how raindrop results in same indexed location•Note how upper point with straight snap would go to wrong catchment•Note how the path does trace the topo delineation for water
Why this is Important?Why this is Important?
Allows for monitoring data to be easily Allows for monitoring data to be easily linked to other types of data (i.e. linked to other types of data (i.e. Impaired waters, assessments, permits, Impaired waters, assessments, permits, etc.)etc.)
Allows for upstream/downstream analysisAllows for upstream/downstream analysis
Greatly expands the usability of Greatly expands the usability of monitoring datamonitoring data
Batch Indexing TimelineBatch Indexing Timeline
This capability should be available This capability should be available January 2009January 2009
Deployed as a service that any data Deployed as a service that any data flow could use (not just for WQX)flow could use (not just for WQX)
Also provide some Geospatial QA Also provide some Geospatial QA services (i.e. is the lat/long in the services (i.e. is the lat/long in the county/HUC/state that the user county/HUC/state that the user specified)specified)
Where to Get StartedWhere to Get Started
1.1. Figure out what data you have, and Figure out what data you have, and what format it is in.what format it is in.
2.2. Find out where you stand with the Find out where you stand with the exchange networkexchange network
3.3. Visit: www.epa.gov/storetVisit: www.epa.gov/storet
4.4. Give us a call, we can walk you Give us a call, we can walk you through any questions.through any questions.
Web ResourcesWeb Resources
www.epa.gov/storetwww.epa.gov/storet www.exchangenetwork.net/exchanwww.exchangenetwork.net/exchan
ges/water/wqx.htmges/water/wqx.htm www.epa.gov/storet/wqx.htmlwww.epa.gov/storet/wqx.html
Question/CommentsQuestion/Comments
Contact:Contact: Dwane Young, EPA (202)566-0616Dwane Young, EPA (202)566-0616
[email protected]@epa.gov
Kristen Gunthardt, EPA (202)566-1194Kristen Gunthardt, EPA (202)[email protected]@epa.gov
STORET Help Desk 800-424-9067STORET Help Desk [email protected]@epa.gov
Join the STORET list serve for more Join the STORET list serve for more information, including monthly conference information, including monthly conference calls about STORET and WQX: calls about STORET and WQX: www.epa.gov/storet/listserv.htmlwww.epa.gov/storet/listserv.html