Development, implementation, and application of an improved model performance evaluation and diagnostics approach

Byeong-Uk Kim*, William Vizuete, and Harvey E. Jeffries

Department of Environmental Sciences & Engineering, University of North Carolina at Chapel Hill

*Georgia Department of Natural Resources

5th Annual CMAS ConferenceOctober 17, 2006

Typical SIP modelingBase case

Future case

Future control case

Base case emissions

Proposed controls

Preset controls

Base case meteorology

Future controlcase emissions

Future projected emissions

Preset controls

= +

More controls until passing attainment demonstration

Model Performance Evaluation

If attainmentdemonstrationfailed

Model performance evaluation (MPE) is the process for assessing the “reliability of model predictions.”

Future caseemissions

Issues with typical MPE practice

MPE for only (if not, mostly) ozone signals No systematic evaluation about if models get right answers

for likely right reasons No evaluation of winds with respect to chemical signals

“Waterfall” procedures and no explicit consideration of the impact of model performance on policy choices

No further MPE for model inputs/outputs with respect to proposed policy options once a MPE is done by following the EPA guidance literally

Probable diagnostic evaluation after many ad hoc analyses

Over-dependence on statistical tests No acceptance for partially useful modeling results No systematic analysis for graphical measures

Needs for investigation of possible causes of poor performance

The expected outcomes of MPE

Is the formulation of a model scientifically acceptable in general? (i.e. what is the adequacy and quality of model formulation for this use?) Concerning if models simulate general causes

Does a model replicate the observations adequately? (i.e. does it make predictions that match history?) Examining if models get right answers for right reasons

Is a model usable for answering specific (e.g. policy) questions? (i.e. does the model fulfill the designed task?) Assessing if models are usable for target purposes

Modified from the original questions in Beck, 2002

Protocol for Regulatory Ozone Modeling Performance Tests (PROMPT)

PROMPT is a meta-protocol; it is a protocol for protocols.Four phases of evaluation procedures

Does this model show or have all necessary components to produce the phenomena that I can expect from the current best perceptual/conceptual model? (Evaluation Phase One)

Can this model distinguish which precursor(s) to control for ozone reduction? (Evaluation Phase Two)

How precisely can the model estimate control requirements? (Evaluation Phase Three)

What are the possible biases in the prediction and the impact of biases on the policy choice? (Evaluation Phase Four)

Performance measures will be examined in a “progressive” manner.

In later evaluation phases, more information will be investigated than earlier phases of evaluation.

PROMPT emphasizes “day-by-day” and “site-by-site” performance analyses and requires evaluators to examine meteorological inputs, ozone, NOx and VOCs as well as geographical features.

Importance of consideration of control options in MPE

Given two winds, A and B, control options for R can be evaluated if the target emission source is the grey area.

Assuming the emission intensity in the grey area is homogeneous in time and space

Illustration of PROMPT application

Houston-Galveston-Brazoria 8-hour Ozone SIP Modeling (“base1b” was used for this example although “base1c” is the newest case)

Modeling period: 2000-08-16 ~ 2000-09-06 Extensive observational data available through TexAQS 2000

campaign Almost same period of Houston-Galveston Mid-Course Review (MCR)

modeling for the 1-hour ozone (“base5b”) In general, this episode shows a very Houston-specific ozone problem;

Transient High Ozone Events (THOEs) that are often characterized by hourly ozone concentration changes more or equal than 40 ppb.

THOEs are often caused by epidemic highly reactive volatile organic compounds (HRVOCs) emission events under ozone-conducive conditions.

No official HRVOC emission event record available for 2000 The possible existence of event emissions in 2000 can be inferred

from a study conducted by UT researchers.

Transient High Ozone Event in Houston

>10,000 lbs/hr ethylene release at La Porte, (6700 lbs between 11:00 AM and 11:25 AM) 3/27/2002

Modeling domain

36 km 12 km 4 km 1 km 1 km

Galveston Bay

Ship Channel

36 km

Modeling with improved model inputs

Issues in base5b (1-hour MCR): Poor surface wind predictions PBL height and vertical mixing Over-predictions in NOx, CO, and HRVOCs at surface

monitors Insufficient ozone formation

Major changes in inputs Meteorology, Emissions, Chemistry, Boundary conditions Yet, questionable 4-km grid resolution

Does the set of new inputs make base1b (8-Does the set of new inputs make base1b (8-hour SIP) more “useful” for assisting hour SIP) more “useful” for assisting decision makers in choosing control options decision makers in choosing control options between NOx control and VOCs control (or between NOx control and VOCs control (or both)?both)?

RegEvnt1 base5b with CMAQ

base5b (1km) base5b

Older 1-h model Newer 8-h model

Base1b Vertical Kv Profile

0

100

200

300

400

500

600

700

800

900

1000

0 50 100 150 200 250 300 350 400

Kv

CA

Mx L

ayer

(m)

(020:029) (021:029) (020:030) (021:030)

Base5b Vertical Kv Profile

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

0 50 100 150 200 250 300 350 400

Kv

CA

Mx L

ayer

(m)

(020:029) (021:029) (020:030) (021:030)

Base1b (8-hr SIP)

Base5b (MCR)Clinton (C35C)

NOx emissions

Bush Airport Only 6AM~5PMDecrease ~4 ppb/h downtown/west Houston; increase perimeter counties

CO emissions

Decrease ~50 ppb in downtown/west Houston

ETH emissions

Only 7AM on 25, 29, and 31Some increases in downtown/west Houston

NO2 Barchart and time series

Drops to model boundary condition at night.

Level 4 in modelCO timeseries

ISOP timeseries

ETH timeseries

O3 peak

8/25

Aloft NO2 and CO

base1b

base5b

Summary of Process Analysis

Overall, OH distribution of reaction with NO2, CO, CH4 ranges 41%~ 48%; very similar new OH radical source strength across HG domain

This is somewhat low compared to other PA results in other areas.

A significant portion of the total OH reaction (=new OH x chain length) is with NO2, CO, CH4, and other non-NO oxidizing paths. The absolutely maximum amount of O3 that can be formed at the four sites ranged from 127 ppb to 150 ppb minus the emitted NO which ranged from 22 to 123 ppb, thus limiting chemical ozone to values between 36 and 103 ppb of ozone.Thus, the chemical production of O3 is inversely proportional to the NOx at these four sites.PAN is predicted to be very low at these sites, so is RNO3.

•What are the implications from What are the implications from insufficient radical source?insufficient radical source?

–The deficient radical sources result in insensitivity to VOC precursors and inhibited due to elevated levels of NOx.–With current model configuration, VOC controls will have little to no effect in future control strategies.

Major SIP-related question

AcknowledgementThe Houston Advanced Research Center and the 8-hour ozone Coalition Group

Texas Commission on Environmental Quality: Dr. Jim Smith for base1b and base5b files

University of Houston: Dr. Daewon Byun and Dr. Soontae Kim for Q20 files

Missing FORM?

Observational Evidence

Two potential sources of HCHOFlares

98-99% combustion assumed, 1% to 2% emitted VOC composition is assumed same as that fed to flare; rest assumed to be CO2. We assumed that HCHO emitted was equal to VOC emitted.

“Flare case” - Assumed that VOCs fed to flares were partially converted to HCHO and that an amount equal to another 1% was emitted as HCHO. This added a total of 55, 58, and 59 tons on 25th, 30th and 31st. to 13 flares located mainly in the eastern part of Houston

Mobile sources New data (SWRI, 2005) on Heavy Duty Diesel show that HCHO is 23%

of VOC and ethene is 18% of THC. HCHO was 5% of CO. We added HCHO at 4% of low level CO

“Mobile case” - Based on AC obs, assumed that MV emissions did not have enough HCHO. An appropriate factor appeared to be 4% of CO. This added 167, 156, and 145 tons on 25, 30, and 31.

DeltaOzone,ppbO8/2513-16hFlare case

DeltaOzone,ppbO8/2509-12hMobile case

SummaryFlare imputation caused >30 ppb increase in ozone concentrationsCO ratio caused >18 ppb increase ozone concentrations, more distributedIncreased peak ozone at almost every monitor causing 4 monitors to match observations~20% increase in new OH and ~30% in ozone productionStill did not match observed HCHO.

OH + (VOCs + CO + CH4 + NO2)

NO225%

CO19%

FORM14%

ALD210%

MGLY1%

OPEN0%

CH44%

PAR8%

MEOH1%

ETOH0%

ETH3%

OLE2%

TOL1%

XYL2%

CRES1%

ISOP

ISPD4%

25 Aug. 2000Bayland Park

Total OH reacted:105.29 ppb

VOC + CO + CH4:75.9 ppb

VOC:52.63 ppb

TCEQ b1b.psito2n2base1b