data assembly center (dac) example data quality procedures vrco discussions, 06 december 2006

Data Assembly Center (DAC)Example Data Quality Procedures

VRCO Discussions, 06 December 2006

The Mission of the National Data Buoy Center (NDBC) is to Deliver Real-Time Meteorological and Oceanographic Data to Weather Forecasting Offices

Real-Time is generally considered < 2 hours after measurement acquisition (< 6 hr, NDBC receives credit)

The NDBC is currently quality controlling nearly 500 remote marine stations (buoys and fixed-stations) and number is growing

Data Management and Communications Branch Status

11 September 2006Weekly Platform Count

0

50

100

150

200

250

300

350

400

450

500

Date

Nu

mb

er

of

Pla

tfo

rms

NDBC DART IOOS NOS O&G TAO Total

Generalized Data Flow

Automated Real-Time QC isperformed on incoming data atthe National Weather Service Telecommunications Gateway. Limited algorithms are applied (range checks and time continuity).

DAC Watch-stander Schedules

Data Assembly CenterDay Watch Routine (Weekday)

I:\\DAC\Procedures\Daywatch Routine.xls

Date: _______________________________ Analyst: __________________________________

Time   Task Report Initials Remarks

6:00   Watch Turnover      

6:15   Review Synoptic situation and Yosimite displays      

6:30   WEB, Yosimite and Iridium Checks      

7:00   Evaluate Location Report      

7:30   Review Brief, add late changes Final Brief    

8:00  Post Briefing Slides to web and send 5-Day Forecast Report

5-Day Forecast Report

8:30   WEB, Yosimite and Iridium Checks      

9:00   Evaluate DAPS reports      

9:30   Perform DART review on web      

10:00   Analyze IOOS/NOS metdiffs (12z-12z)      

10:30   WEB, Yosimite and Iridium Checks      

11:00   Analyze IOOS/NOS metdiffs (12z-12z)    

11:30          

12:00   Analyze IOOS/NOS plots IOOS/NOS Daily

Update   

12:30   WEB, Yosimite and Iridium Checks      

13:00   Review Wave ValidationWave Validation

Report   

14:00   Analyze DART Summary Report      

14:30   WEB, Yosimite and Iridium Checks      

15:00   Evaluation Long Island Time Tab      

15:30   Review Critical BuoysCritical Buoy

Report   

16:00   Evaluate Communication Summary      

16:30   WEB, Yosimite and Iridium Checks      

17:00   Review Buoys no reporting No Data Report    

17:30   Prepare for Turnover      

18:00   Turnover      

Data Assembly CenterNight Watch Routine (Weekday)

I:\\DAC\Procedures\Nightwatch Routine.xls

Date: _______________________________ Analyst: __________________________________

Time   Task Report Initials Remarks

18:00   Watch Turnover      

18:15   Review Synoptic situation and Yosimite displays      

18:30   WEB, Yosimite and Iridium Checks      

18:45   AIS check      

19:00   Evaluate Location Report      

19:30   Preliminary Briefing Slides Preparations      

20:00   Analyze Power Data      

20:30   WEB, Yosimite and Iridium Checks      

21:00          

21:30   Perform DART review on web      

22:00   Analyze NDBC metdiffs (00z-00z)      

22:30          

23:00   WEB, Yosimite and Iridium Checks      

23:30   Analyze NDBC plots NDBC QC Update    

0:00          

0:30   WEB, Yosimite and Iridium Checks      

1:00   Evaluate Communications Summary      

1:30  Evaluate Station Location Report and prepare Drifting Buoy Report

Drifting Buoy Report    

2:00   Analyze DART Summary Report      

2:30   WEB, Yosimite and Iridium Checks      

3:00          

3:30   Prepare Briefing Slides      

4:00          

4:30   WEB, Yosimite and Iridium Checks      

5:00   Finalize Briefing Slides      

5:30   Prepare for Turnover      

6:00   Turnover      

Manual (DAC Watch-stander)Real-Time QCMarta Systems (Yosemite WeatherWorkstation)

•Wind barbs•Yellow circles indicate comms loss

(data > 2 hours old)•Overlaid satellite stitched graphics

and looping

Watch-standers must actively look at Screen – Not as many screens asGeographical Regions

Marta Display – Overall View

Marta Display – Gulf of Mexico

Near-Real-time QC(After Automated Real-Time QC)

•Intensive Database Set-up•ASCII IDs Associated with

measured parameters•Analysts run programs to search for

anomalous data•Program output on line printers

•DAC Analysts set-up all stations indatabase

Oracle Database – NEMIS GUI Front-end

Oracle Database – ASCII ID Assignment

Example METDIFF Printout Page

Lower-case flags, warning-onlyDoes not stop data

Generates stacks of paper

Example METDIFF Printout Page‘D-Flagged’ Data

Upper-case FlagsData flow is terminateduntil released by analyst

Example METDIFF Plots Region

Sta

tio

ns

Dat

a D

isp

lays

Oracle Database – Raw Data Report

Oracle Database – Raw Data Report

Oracle Database – Interactive Plotting

Communications Report

•Looking for data dropouts dueto communication problems

•Analyst runs program and scansoutput

Oracle Database – Communications Report

“M” – Missing Data (Comms)

Buoy Location Report

•Looking for buoys off-station“drifting buoys”

•Analyst runs program and scansoutput

Oracle Database – Buoy Location Report

% of Watch circle>100% is drifting