ihop-2002 data archive and development of composite data sets steven f. williams, scot m. loehrer,...
TRANSCRIPT
![Page 1: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/1.jpg)
IHOP-2002 Data Archive and Development of Composite Data Sets
Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine
Goldstein, and Don Stott
UCAR/Joint Office for Science SupportIHOP-2002 Spring Science Workshop
Boulder, CO March 2003
![Page 2: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/2.jpg)
http://www.joss.ucar.edu/ihop/dm/
![Page 3: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/3.jpg)
IHOP-2002 Data Policy and Protocol
![Page 4: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/4.jpg)
![Page 5: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/5.jpg)
http://www.joss.ucar.edu/ihop/dm/photo.html
![Page 6: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/6.jpg)
IHOP-2002 DATA BASE STATUS(as of 24 March 2003)
• 196 of 254 Data sets (77%) have been submitted or linked. 183 are available on-line (13 in progress)
• Field Catalog (w/reports and preliminary products) is on-line
• Data submission instructions and guidelines are available on-line
• For details see: http://www.joss.ucar.edu/ihop/dm/
![Page 7: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/7.jpg)
Composites: What and Why
A composite dataset is a collection (over some time period and region) of similar data (e.g. surface met) from a variety of sources, put into a common format, and passed through a uniform quality control.
Why does UCAR/JOSS develop composites?
- Provides data in a uniform format with QC.
- Allows determination of network/site problems.
- Useful for model applications.
- Prevents duplication of effort.
![Page 8: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/8.jpg)
Composite Data Sets To Be Developed by UCAR/JOSS for IHOP-2002
• Surface Meteorological - 1 min, 5 min, and Hourly
• Precipitation - 15 min, Hourly, and Daily
• Soundings - High Resolution and
5 hPa
![Page 9: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/9.jpg)
1-min Surface Meteorological Data Composite (65 stations)
ASOS (41)
ARM SMOS (14)
ABLE AWS (5)
NCAR Supp. (5)
![Page 10: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/10.jpg)
Oklahoma Mesonet (115)
NCAR ISS/ISFF (10)
West Texas Mesonet (34)
1-min sites (65)
5-min Surface Meteorological Data Composite (224 stations)
![Page 11: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/11.jpg)
Hourly Surface Meteorological Data Composite (743 stations)
CoAgMet (*) (23)
AWOS (*) (110)
RAWS (+) (10)
SCAN (o) (6)
GWMD (*) (18)
Ameriflux (+) (3)
HPCN (o) (72)
Texas ET (o) (19)
KVII-TV (+) (4)
MADIS (o) (246)
NMSU (*) (7)
5-min sites (o) (225)
![Page 12: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/12.jpg)
15-minute Precipitation Composite
NCDC Coop (429)
5-min surface (224)
![Page 13: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/13.jpg)
Hourly Precipitation Composite
NCDC Coop (509)
NCEP hourly (1155)
Hourly Met (743)
![Page 14: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/14.jpg)
Daily Precipitation Composite
NCDC Coop (1581)
Hourly Met (743)
NCEP Daily (1611)
![Page 15: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/15.jpg)
GENERAL DATASET PROCESSING LIFE CYCLE
Contact SRC Agency to Acquire Data and Metadata
Wait for SRC Agency Data/Metadata Preparation
Examine Data & Metadata for Consistency and Completeness
Acquire Data and Metadata OR Create “Best” Metadata
Develop New Software or Modify Existing Software
Acquire Computer System Space and Time
Process Data and Metadata then Check
Form Composite with Like Datasets
Perform QC and Final Data Checks. Generate Statistics
Create EBUFR format data & DOCS for CODIAC
Load Data and Metadata onto CODIAC
ACTION
Prepare ASCii format data & docs for CD-Roms
Prepare, manufacture, & distribute CD-Roms
Add stn info to final list
Incomplete?
ProcessingProblems?
1
2
z:\cully\Esop96\esop3.ppt
Incomplete?
Incomplete?
5
6
7
8
9
3
4
Tim
e
• Phone, email, paperwork, $, meetings, procurement arrangements. (Days to months)
• (Minutes to months)
• Auto retrieval, FTP, online requests, etc. (minutes to days)
• Verify AOI, TOI, Format; spot check values, visual inspection or limited processing (Minutes to days)
• Format change, new parameters manual “cleanup”, etc. (Days to months)
• Purchase hardware, configuration management, etc. (Minutes to days)
• Put into common format, run data checkers “check gross limits” (Days to months)
• HQC for SFC, Gross limits for precip, Composite level and value checkers, limited statistics (% G, B, D by network, stn, parameter for SFC). (days to months)
da
ys
/mo
nth
sw
ee
ks
/mo
nth
sm
on
ths
/ye
ar
![Page 16: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/16.jpg)
QC of Surface Meteorological Data
• Parameters: Calculated Sea Level Pressure, Sea Level Pressure, Temperature, Dew Point, Wind Speed, Wind Direction
• Expected value derived from objective analysis method adapted from Cressman (1959) and Barnes (1964) using 300km Radius of Influence.
Σθiwi
Σwi
i
i
θe = Expected Value
θi = Observation at site i
wi = Weight factor for site i where wi = 1/di
di = Distance between site i and QC site
θe =
![Page 17: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/17.jpg)
QC of Surface Meteorological Data • Deviation between actual and expected values subjected
to dynamically determined limits. Values compared at like solar times.
Limits are sensitive to diurnal and intra-seasonal variations and dependent upon spatial and temporal continuity.
• Additional consistency checks, visual quality assurance, and gross limit checks are typically also applied.
• Precipitation gross limit checked based on frequency.• Data are never changed, only flagged.
α = Normalization Factor (predetermined)
σ = Standard Deviation (computed over previous 30 days for each time; 15 obs minimum)
θa - θe ασ
![Page 18: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/18.jpg)
What Can Lead to Surface Data Being Flagged?
• Instrument failure, bias, drift, random errors• Incorrect latitude, longitude, or elevation• Differences in siting or height of instruments• Misidentified units, dates, or stations• Terrain impacts• Data problems at surrounding stations• Unique meteorological phenomena
![Page 19: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/19.jpg)
Obvious Problems Detected
![Page 20: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/20.jpg)
Subtle Problems Detected
![Page 21: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/21.jpg)
Embedded Accumulative Precipitation
![Page 22: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/22.jpg)
High Resolution and 5 hPa Radiosonde Composites
Fixed Sites: Mobile Sites:
NWS (14) ARM (5) ISS (1) Falcon Drops (86) Learjet (330) NCAR GLASS (146) NSSL CLASS
(80)
![Page 23: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/23.jpg)
UCAR/JOSS Atmospheric Sounding Processing Procedures
Observing System Output e.g. NWS Micro-ART
Convert to JOSS Quality-control Calculate Derived
Format (ASCII) Parameters
Vertical Consistency Checks
QC Flags
Gross Limit Checks
Automated Checks QC Flags
e.g., u, v, dz/dt
Visual Examination
Error Files
High-resolution and/or Interpolated Vertical Composite e.g., 5 hPa
Inventory, Archive, and Develop Metadata
Data and Metadata Available Online
![Page 24: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/24.jpg)
Wind Oscillations
• Wind oscillations at low elevation angles (approaching limiting angles) due in part to questionable angles.
Original Algorithm
Adjusted Algorithm
![Page 25: IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,](https://reader036.vdocuments.net/reader036/viewer/2022062315/5697bf871a28abf838c888a4/html5/thumbnails/25.jpg)
Summary
• The High Resolution and 5 hPa Radiosonde Composites have been completed and are available: http://www.joss.ucar.edu/ihop/dm
• Surface Meteorological (1-min, 5-min, and hourly) and Precipitation (15-min, hourly, and daily) Composites currently being developed. Planned completion Fall 2003.