new tools for a better air quality system data flow en 2012 may 31, 2012
TRANSCRIPT
New Tools for a BetterAir Quality System Data Flow
EN 2012May 31, 2012
• Air Quality System (AQS) flow automation enhancements
• Exchange Network Services Center• AQS Plugins for OpenNode2 and EN-Node• Q&A / Discussion
Topics to Cover
• AQS submissions via a Node or Node Client used to require extra manual steps
• More difficult to access AQS submission logs and reports
• Most users preferred to use EPA’s non-Exchange Network interface called CDX-Web
Jumping Through Hoops
Can I interest you in an AQS data flow?
• Today, files submitted through the Exchange Network are automatically processed by AQS
• Exchange Network Nodes can download the submission log and reports
• New tools for submitting AQS data via the Network• Support for AQS submissions through CDX Web is being
discontinued
Fewer Flaming Hoops…More Puppies
Perhaps this is more to your liking?
• AQS data flow (v2.2a) now uses the Exchange Network Header
• The Header file contains information about the submission data file– Security Token– Exchange Network and AQS User IDs– Payload type: 1) XML or 2) FLAT– AQS Load Options
• Final Processing Step• Stop on Error
How Does it Work?
The submitter defines the desired level of automation:
1. Stage: Submission file is uploaded to the AQS server and verified to be a valid AQS file type, but with no subsequent processing.
2. Load: The file is loaded into the AQS database and its contents validated. Sample measurements are left at the status “Pre-production”; all other data types (Site information, Monitor information, and QA information) are set to the status “Production”.
3. Post: All valid sample measurement data for the file is set to the status “Production”.
– Post is the default setting
AQS Load Options
• Exchange Network Services Centerhttps://enservices.epa.gov/login.aspx –Automatically creates EN Header–Supports manual submissions to AQS–Replacing CDX-Web for AQS submissions
New Submission Option #1
Node Plugins for OpenNode2 and EN-Node
•Customized for easy integration with 2 widely used air monitoring systems
– OpenNode2 - .NET – AirVision and DR-DAS– OpenNode2 - Java – DR-DAS– EN-Node - .NET – AirVision– EN-Node – Java - AirVision
•Automatically create EN Header•Support submissions to AQS and data publishing•In testing. Production-ready versions likely available in June for free download from EN website.
New Submission Option #2
Show and Tell
• Exchange Network Services Center
• EN-Node Plugin– AirVision
• OpenNode2 Plugin– AirVision & DR-DAS
EN-Node AirVision Plugin
• AirVision Plugin Features 3 Operations Submit AQS data to CDX Publish AQS data (Solicit) Publish AQS data (Query)
1
2
3
EN-NodeAQS
Monitor
AirVision
AirVisionPlugin
ENDS
CDXEPA AQS
2
1
3
Submit
SolicitQueryEN Browser
• URL: http://code.google.com/p/en-node2/downloads/list• File: AirVisionPlugIn.zip
Installation File & Contents
File Name Purpose Installation1. Six (6) DLL files Plugin software Copy to EN-Node
file directory
2. InsertAQSHeaderTemplate.sql DB scripts to install AirVision Plugin in EN-Node DB
Execute scripts
3. Task_SubmitAQSXMLData_DataWizardConfig.xml
Submit AQS data to CDX
Copy to EN-Node file directory
4. Query_GetAQSXMLData_DataWizardConfig.xml Publish AQS data (Query)
Copy to EN-Node file directory
5. Solicit_GetAQSXMLData_DataWizardConfig.xml Publish AQS data (Solicit)
Copy to EN-Node file directory
• What is Configuration?– Business logics used by EN-Node to complete the data flow – Config. data saved in an external BEPL file (Business Process Execution Language) – Each Operation has its own BEPL file – Configuration tools
• EN-Node Admin Console• BEPL XML file provided with the Plugin installation file
• How to Configure:Step 1: Create a Domain (AirVision)Step 2: Register 3 operations under AirVision1. Submit AQS data to EPA2. Publish AQS (Query)3. Publish AQS (Solicit)
Step 3: Test PluginStep 4: Publish AQS data at ENDS or EN Browser
Configure AirVision Plugin
Plugin Configuration Options
Data Needed to Configure AirVision Plugin
Parameter Required? Value Example
EPAURL Yes URL of the EPA Endpoint in which to submit data
https://testcdxnode64.epacdxnode.net/cdx-enws20/services/NetworkNode2
NAASUserID Yes User Name EnfotechNAASUserPassword Yes Password enfopassword
AirVisionURL Yes URL of the AirVision Node http://localserver/AirVision.WebService
FileName No File Name Airvision_Submission
AQSXMLSchemaVersion No Version Number 2.2
CompressPayload No True/False TrueDurationCode No Number in hours 1
StartTime Yes Date in the format YYYY-MM-DD 2011-10-01
EndTime Yes Date in the format YYYY-MM-DD 2011-12-31
SendOnlyQAData No True/False FalseSendRATransactions No True/False TrueSendRBTransactions No True/False TrueSendRDTransactions No True/False TrueSendRPTransactions No True/False True AuthorName No Author Name John SmithOrganizationName No Organization Name Org1 Inc.DocumentTitle No Document Title AQS SampleAQSUserID No AQS User Identifier EnfotechAQSScreeningGroup No AQS Screening Group EnfotechAQSSchemaVersion No Schema version of document 2.0
AQSPayloadTypeProperty No Format in which data is presented XML
AQSLoadOptions No Load/Operation LoadPayloadOperation No Load/Operation LoadAgencyCode No Agency Identification Code 001SiteCode No Site Identification Code 210ParameterCode No Parameter Code 46324
• AirVision End Point• EPA AQS End Point• Organization Name• NAAS ID & PWD• Plugin file location
Minimum Data Needed
Before Configuration…
Important!• Open 3 DataWizardCinfig.xml files with text editor• Enter Plugin file location
• Create AirVision domain – Choose web services (Submit, Download, Query, Solicit, Notify) allowed for
the domain.
• Establish “Administrator” accounts to setup/modify this domain.
Step 1: Create Domain
• Register Operation(s)1. Submit AQS data to EPA: Task_SubmitAQSXMLData_DataWizardConfig.xml2. Publish AQS (Query): Query_GetAQSXMLData_DataWizardConfig.xml 3. Publish AQS (Solicit): Solicit_GetAQSXMLData_DataWizardConfig.xml
Step 2: Register Operation(s)
Define•EPA Endpoint URL•Validation rule (if any)•Data viewing stylesheet (if any)
Define•Task schedule
• Register Operation(s)1. Submit AQS data to EPA: Task_SubmitAQSXMLData_DataWizardConfig.xml2. Publish AQS (Query): Query_GetAQSXMLData_DataWizardConfig.xml 3. Publish AQS (Solicit): Solicit_GetAQSXMLData_DataWizardConfig.xml
Step 2: Register Operation(s)
1. Test Submit AQS to EPA– Test: Enter Start Date and End Date to a
date earlier than the current date will execute task after the user clicks save.
– Track submission status CDX will return “Completed” or “Failed” View Download Data Viewer
Step 3: Test Operation(s)
Authentication
Query Request
2. Test AQS Data Query– Authenticate – Input Start Date and End Date for AQS
monitoring date– Query will return data in XML
Step 3: Test Operation(s)
Authentication
Solicit Request
Get Status
Call Download
3. Test AQS Solicit– Authenticate – Solicit (Input AQS monitoring date)– Get Status– Download AQS data (in XML)
Step 3: Test Operation(s)
• Manual data input to ENDS:– Test: https://ends2.epacdxnode.net– Production: https://ends2.epa.gov
• XML Data Upload to ENDS:– Auto Pull from ENDS: GetServices – Manual XML Upload to ENDS
Step 4: Publish AQS Data at ENDS/ENB
ENDS: Service/DEDL Registration
Option A: Manual
Option B: Automated
(Published via GetServices)
(Node submits DEDL to EPA)DEDL XML
ENDS XML
Node Admin
Data Provider
Exchange Network
Node
EPA Service Registry (ENDS)
(Automate your Node to transfer registration info to EPA’s Registry)
Stores registry of all services, input parameters, pick lists, etc.
B1
B2
Node Admin Registers Services (ENDS)
Node Admin Registers DEDL
A1
A2
EN Browser
• Test: https://ends2.epacdxnode.net/Node2WS.svc • Production: https://ends2.epa.gov/Node2WS.svc • Dataflow: ENDS_v20• Operation: Service Refresh
• Implement Only the Operation(s) you need• Minimum data you need to configure Plugin
– AirVision End Point– EPA AQS End Point– Organization Name– NAAS ID & PWD– Plugin file location
• Useful resources– Plugin: URL: http://code.google.com/p/en-node2/downloads/list– Data Publication Guide:
http://www.enbrowser.net/enbrowser/ClientBin/HowToPublishData.pdf – ENDS & FCD: http://www.exchangenetwork.net/2011/07/15/exchange-
network-discovery-service-ends/
• Support– enfoTech, Lijie Xu: [email protected]
Useful Tips for EN-Node AirVision Plugin
Show and Tell:OpenNode2 AQS v2.2 Plugin
• http://code.google.com/p/opennode2• .NET OpenNode2
– AirVision – DrDAS
• Java OpenNode2– DrDAS
OpenNode2 AQS v2.2a Plugins
OpenNode2 AQS Submit Workflow
OpenNode2
AirVisionOr
DrDASWeb
Service
AirVisionOr
DrDASWeb
Service
EPA CDX
Node
EPA CDX
Node
1. Request Data
2. Return Data
3. Submit Data
1. On a set schedule (or manually triggered), request data from AirVision or DrDAS• Parameters set to filter data request
2. AQS XML data is returned to OpenNode23. Relay the data to EPA CDX/AQS system
• Optionally, header is added• Optionally, notifications can be sent
Get Status/Download Workflow
OpenNode2
EPA CDXEPA CDX
Status Response1. Get Status
Files2. Download Request
3. Notifications
1. On a set schedule (or manually triggered), check status of outstanding submissions
2. If CDX returns “completed” or “failed”, download processing reports from CDX• Do nothing if status is “pending”
3. Optionally, send notification emails to agency staff1. Mail sent for all submissions where status
changes from “pending” to “completed” or “failed”
• “Publishing” makes data available for any authorized partner to come and query from your node– Exposed as Query and/or Solicit services
• No additional setup needed to make these services available from OpenNode2– Setting up the services in Node Admin tool makes
them available with a simple checkbox
Publishing AQS Data from OpenNode2
Setting up the AQS Plugin/Flow
• Very similar to setting up any other OpenNode2 data flow
• Using the Node Admin interface:1. Create the AQS flow2. Upload the plugin (contains all workflow logic)
3. Configure the Services (exposes plugin logic)a) Service 1: Get data from AirVision/DrDAS for submission to EPAb) Service 2: Check status of submission and download results
4. Create the Schedules (automates workflow)
… and that’s it for the OpenNode2 portion
OpenNode2 AQS Service Setup
Flow 1Service 1Service 2
Flow 2Service 1Service 2
View in Node Admin of AQS flow with services configured:
AQS “GetXmlData” Service Detail
Note: AirVision settings shown DrDAS settings very similar
AirVision Schedule Detail
DrDAS Schedule Detail
Parameter Value
FileGenerationPurposeCode “AQS”
BeginDate Desired monitoring period start date
BeginTime “00:00”
EndDate Desired monitoring period end date
EndTime “23:59”
DataValidityCode “V”
DataApprovalIndicator “Y”
IncludeMonitorDetails “N”
IncludeEventData “FALSE”
SchemaVersion “2.2”
Kurt [email protected]
Questions?