1

A Data Management System (DMS) for A Data Management System (DMS) for Air Quality Management AgenciesAir Quality Management Agencies

Prepared by:Prepared by:Mark StoeltingMark Stoelting

Bay Area Air Quality Management DistrictBay Area Air Quality Management DistrictSan Francisco, CASan Francisco, CA

andandEric GrayEric Gray

Sonoma Technology, Inc.Sonoma Technology, Inc.Petaluma, CAPetaluma, CA

Presented at:Presented at:U.S. EPA 2008 National Air Quality ConferenceU.S. EPA 2008 National Air Quality Conference

Portland, ORPortland, ORApril 6-9, 2008April 6-9, 2008

2

DMS: Why, What, How, WhoDMS: Why, What, How, Who

• Why build a DMS?Why build a DMS?

• What is it?What is it?

• How does DMS work?How does DMS work?

• What are the DMS capabilities?What are the DMS capabilities?

• Who is involved?Who is involved?

• Who paid for DMS?Who paid for DMS?

• Where is it going?Where is it going?

3

Why the DMS?Why the DMS?

• New technology: instruments, New technology: instruments, communications, data accesscommunications, data access

• Real-time data accessReal-time data access

• Improve data quality (QC checks)Improve data quality (QC checks)

• Graphical data reviewing toolsGraphical data reviewing tools

• Minimize/eliminate need for paperMinimize/eliminate need for paper

• Track instruments and performanceTrack instruments and performance

• Shared need by all monitoring agencies Shared need by all monitoring agencies

4

What is DMS?What is DMS?

• Data Management SystemData Management System• Database to ingest, process, export dataDatabase to ingest, process, export data• ArchiveArchive

• Programmable database with Programmable database with optionaloptional featuresfeatures• Process sub-hourly data (1 minute)Process sub-hourly data (1 minute)• Collect instrument metadataCollect instrument metadata• Track instruments and performance (OpCodes)Track instruments and performance (OpCodes)• Automatic Quality ChecksAutomatic Quality Checks• Export (AirNow, AQS)Export (AirNow, AQS)• Notification (email, process controls)Notification (email, process controls)

5

Simple Data StructureSimple Data Structure

DATA

SITE PARAMETER

INSTRUMENT

{OpCode}

Site,Parameter,Date_Time,OpCode,Data,[alpha_data]Site,Parameter,Date_Time,OpCode,Data,[alpha_data]

70701111,CO_chamt,10/05/2007 08:16,0,48.1580,,CO_chamt,10/05/2007 08:16,0,48.1580,

(optional, assigned in database)

DATE_TIME

6

Sites, Parameters, InstrumentsSites, Parameters, InstrumentsEstablish MonitoringSites

Create Parameters

CreateInstruments

7

Assign Parameters to Instruments Assign Parameters to Instruments and Instruments to Sitesand Instruments to Sites

Assign Parametersto Instruments

Assign Instrumentsto Sites

8

Automated Quality Checks/Export

RawData

AmbientAir Quality

DataRangechecks

Instrumentchecks

Site checks

Network (off-site)checks

AIRNow

DistrictWeb Page Exports filtered by QC code

9

Automatic Quality ChecksAutomatic Quality ChecksAssign AutomaticQuality ControlChecks toParameters

10

Data Input ProcessData Input Process• Ingests data files generated by datalogger or Ingests data files generated by datalogger or

data formatting softwaredata formatting software• Duplicate input data is automatically rejectedDuplicate input data is automatically rejected• Automated Quality Checks establish initial Automated Quality Checks establish initial

QC codeQC code• Min/max, zero drift, sticking, rate-of-change, compareMin/max, zero drift, sticking, rate-of-change, compare

• QC conditions can activate process/alarmQC conditions can activate process/alarm• Electronic chain-of-custody createdElectronic chain-of-custody created• Sub-hourly data aggregated to 1-hr averagesSub-hourly data aggregated to 1-hr averages• Automated data export to AIRNow and Air Automated data export to AIRNow and Air

District web siteDistrict web site

11

DMS StatusDMS StatusStation Reporting Status

Event Log

QC Log

12

Manual Data ReviewManual Data Review

• Time-series graphical editing and QC Time-series graphical editing and QC tools for staff data reviewtools for staff data review

• DMS records all reviews and manual DMS records all reviews and manual adjustments to QC status and data adjustments to QC status and data values (electronic chain-of-custody)values (electronic chain-of-custody)

• Tracks manual data reviewer statusTracks manual data reviewer status

• Lock final data prior to exportLock final data prior to export

• Exports final data for regulatory Exports final data for regulatory submission to AQS or other agenciessubmission to AQS or other agencies

13

Time Series Edit/QC ScreenTime Series Edit/QC Screen

14

Scatter Plot AnalysisScatter Plot Analysis

15

Other DMS FeaturesOther DMS Features

• Data access security (user rights Data access security (user rights management)management)

• Maintains instrument equipment Maintains instrument equipment databasedatabase

• Supports electronic instrument and site Supports electronic instrument and site logslogs

• Daily instrument performance reports Daily instrument performance reports (using OpCodes)(using OpCodes)

16

DMS Software RequirementsDMS Software Requirements

• DMS host computer:DMS host computer:• Windows Server 2000/2003 or XP/VistaWindows Server 2000/2003 or XP/Vista• Microsoft SQL Server 2005 (all editions)Microsoft SQL Server 2005 (all editions)

• DMS Client Software:DMS Client Software:• .NET 2.0 Framework.NET 2.0 Framework• Windows 2000, XP, VistaWindows 2000, XP, Vista

• Future tools may require Java and ERSI Future tools may require Java and ERSI arcGISarcGIS

17

DMS HardwareDMS HardwareDMS Server:DMS Server:• Minimum server requirements – Pentium 4 Minimum server requirements – Pentium 4

laptoplaptop• Must support Microsoft WindowsMust support Microsoft Windows• Hardware determined by processing needsHardware determined by processing needs• Laptop processes hundreds of thousands of Laptop processes hundreds of thousands of

records per hour and store millionsrecords per hour and store millions• Server can process millions of records per Server can process millions of records per

hour and store billionshour and store billionsDMS Client:DMS Client:• Pentium III class PC or betterPentium III class PC or better• Broadband connection to DMS ServerBroadband connection to DMS Server

18

DMS Performance BenchmarksDMS Performance Benchmarks

• AIRNow DMC DMS will be running in August AIRNow DMC DMS will be running in August on a Quad Core Intel processor with 8 GB of on a Quad Core Intel processor with 8 GB of RAM. The operating system will be Windows RAM. The operating system will be Windows Server 2003 64-bit EditionServer 2003 64-bit Edition

•• Currently, AIRNow DMC processes 12 million Currently, AIRNow DMC processes 12 million records a day for over 5000 sitesrecords a day for over 5000 sites

•• The DMS system can process 22 million The DMS system can process 22 million records per hour or ½ billion a day (historical records per hour or ½ billion a day (historical data import)data import)

•• The Bay Area DMS system will process 1 The Bay Area DMS system will process 1 million data records per daymillion data records per day

19

DMS Storage BenchmarksDMS Storage Benchmarks

• The database will grow by 100 MB for The database will grow by 100 MB for every one million recordsevery one million records

• One billion records can be stored in 100 One billion records can be stored in 100 GB dataGB data

• DMS query performance has been DMS query performance has been tested with ½ billion data recordstested with ½ billion data records

• AIRNow DMC will test performance for AIRNow DMC will test performance for over a billion records by fallover a billion records by fall

20

DMS Can Be …DMS Can Be …

• An instrument monitoring systemAn instrument monitoring system

• A data analysis systemA data analysis system

• A data Quality Check and Review A data Quality Check and Review systemsystem

• A data export tool (to AirNow, AQS)A data export tool (to AirNow, AQS)

• A data warehouse for historical dataA data warehouse for historical data

21

Agency DeploymentAgency Deployment• DMS developed using AIRNow technology in DMS developed using AIRNow technology in

20042004• DMS to replace AIRNow technology by fall DMS to replace AIRNow technology by fall

20082008• DMS in use by BAAQMD, SCAQMD, and DMS in use by BAAQMD, SCAQMD, and

CARB by end of 2008CARB by end of 2008• AIRNow International will be based on DMS AIRNow International will be based on DMS

for 2009for 2009• Stable DMS distribution package available for Stable DMS distribution package available for

agency use by early 2009agency use by early 2009• Looking for 3 or 4 additional agency Looking for 3 or 4 additional agency

development partners in 2008development partners in 2008

22

Development and FundingDevelopment and Funding• DMS was development and funded by DMS was development and funded by

agencies with an interest in the resultagencies with an interest in the result• Contributors include:Contributors include:

• Bay Area AQMD (BAAQMD)Bay Area AQMD (BAAQMD)• EPA Region IX, 103 GrantEPA Region IX, 103 Grant• EPA National Environmental Information EPA National Environmental Information

Exchange Network (NEIEN) Challenge GrantExchange Network (NEIEN) Challenge Grant• South Coast AQMD (SCAQMD)South Coast AQMD (SCAQMD)• EPA (AirNow and AirNow International)EPA (AirNow and AirNow International)• California Air Resources Board (CARB) soonCalifornia Air Resources Board (CARB) soon

• New DMS features must be available to all New DMS features must be available to all agenciesagencies

• New features must not reduce DMS New features must not reduce DMS functionality to other agenciesfunctionality to other agencies

23

Future DMS DevelopmentFuture DMS Development

• Enhanced view/edit featuresEnhanced view/edit features• Viewing tools - fingerprint plotsViewing tools - fingerprint plots• Statistical tools - box plots, reportsStatistical tools - box plots, reports

• Support for toxics, PM filter data, Support for toxics, PM filter data, Special Purpose MonitoringSpecial Purpose Monitoring

• Expanded reporting capabilities Expanded reporting capabilities

• Instrument Control ChartsInstrument Control Charts

• GIS mapping capability, web services, GIS mapping capability, web services, and NEIEN XML data exchange nodeand NEIEN XML data exchange node

24

DMS SummaryDMS Summary

• Simple, scalable, cost-effectiveSimple, scalable, cost-effective

• Fast, secure, and efficient operationFast, secure, and efficient operation

• Comprehensive quality control toolsComprehensive quality control tools

• Applicable to a wide variety of dataApplicable to a wide variety of data

• Free to public agencies under a share-Free to public agencies under a share-ware use agreementware use agreement

25

Contact InformationContact Information

Mark StoeltingMark Stoelting

Bay Area Air Quality Management DistrictBay Area Air Quality Management District

mstoelting@baaqmd.govmstoelting@baaqmd.gov

(415) 749-4619(415) 749-4619

Eric Gray Eric Gray

Sonoma Technology, Inc.Sonoma Technology, Inc.

eric@sonomatech.comeric@sonomatech.com

(707) 665-9900(707) 665-9900