![Page 1: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/1.jpg)
A PERFORMANCE EVALUATION OF THE ETA - CMAQ
AIR QUALITY FORECAST MODEL
FOR THE SUMMER OF 2004
CMAS WorkshopChapel Hill, NC
20 October, 2004
![Page 2: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/2.jpg)
Contributors
Brian Eder Daiwen KangTanya Otte Hsiu-mu Lin Jon Pleim Shaocai YuGeorge Pouliot STCKen SchereJeff YoungASMDARL, NOAA
All: On assignment to NERL, U.S. Environmental Protection Agency
Disclaimer: “The research presented here was performed under the Memorandum of Understanding between the U.S. Environmental Protection Agency (EPA) and the U.S. Department of Commerce's National Oceanic and Atmospheric Administration (NOAA) and under agreement number DW13921548. Although it has been reviewed by EPA and NOAA and approved for publication, it does not necessarily reflect their policies or views.
![Page 3: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/3.jpg)
Forecast Configuration
- Eta Meteorology- CBIV Mechanism- SMOKE Emissions (Offline)- 12 km grid resolution - 22 Vertical Layers- 48 Hr. Forecast (12Z Init.)
Simulation Period
- 1 May – 30 September 2004- 1 June – 30 September 2004 (presented here)
Domain
Models-3 CMAQ
Lambert Conformal Projection
142
166
![Page 4: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/4.jpg)
This performance evaluation used:
Hourly O3 (ppb) from EPA’s
AIRNOW network:
- over 600 stations (mostly urban)
- for a four month period (J, J, A, S)
A suite of statistical metrics for:
- discrete forecasts (MB, NMB, RMSE, NME)
- categorical forecasts (A, B, POD, FAR, CSI)
of hourly, max. 1-hr and max. 8-hr O3 (presented here).
![Page 5: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/5.jpg)
Discrete Forecast / Evaluation
[Observed] [Observed] versus versus [Forecast] [Forecast]
Statistics
- Summary
- Biases MB
NMB
- Errors
N M EM odel O bs
O bs
N
N
1
1
100%( )
R M SE M odel O bsN
N
1 2
1
0 5
( )
.
![Page 6: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/6.jpg)
Category Forecast / Evaluation
Observed Exceedances, Non-ExceedancesObserved Exceedances, Non-Exceedances versusversus
Forecast Exceedances, Non-ExceedancesForecast Exceedances, Non-Exceedances
a b
c d
Fo
reca
st E
xcee
danc
e
N
o
Yes
No YesObserved Exceedance
a b
c d
Ab c
a b c d%
100
Ba b
b d
FARa
a b%
100
C SIb
a b d
100%
PO Db
b d%
100
![Page 7: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/7.jpg)
Monthly summaries:
- June, July, August, September
- Both Discrete and Categorical evaluation
Daily summaries:
- August 4 -12, Discrete evaluation
- Examine performance over a full synoptic cycle
- Illustrating both good and poor model performance
Forecast / Evaluation
![Page 8: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/8.jpg)
Meteorological Conditions for the Summer (J, J, A) 2004
![Page 9: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/9.jpg)
Meteorological Conditions for the Summer (J, J, A) 2004
Temperature Precipitation
![Page 10: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/10.jpg)
Monthly Summaries
![Page 11: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/11.jpg)
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
46.1 53.9 0.51 7.8 16.9 14.1 24.0
A B POD FAR CSI a=40 b=19
99.5 0.79 25.3 67.8 16.5 c=18274 d=56
a b
c d
June Summary
![Page 12: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/12.jpg)
Correlation Mean Bias (ppb)
NMB (%) NME (%)
June Summary
![Page 13: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/13.jpg)
Domain Wide Daily Max. 8- Hour Means
0
10
20
30
40
50
60
70
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29
June
(pp
b)
OBS_MEAN MOD_MEAN
June Summary
1 15 30
![Page 14: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/14.jpg)
June Summary
Bias
Error
1 15 30
![Page 15: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/15.jpg)
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
47.4 57.3 0.55 9.9 20.8 16.2 27.2
A B POD FAR CSI a=272 b=95
98.0 1.9 50.5 74.1 20.7 c=18258 d=93
July Summary
![Page 16: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/16.jpg)
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
43.4 55.1 0.62 11.7 27.1 16.5 30.9
A B POD FAR CSI a=232 b=19
98.7 5.2 39.6 92.4 6.8 c=19000 d=29
August Summary
![Page 17: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/17.jpg)
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
41.7 52.8 0.65 11.1 26.7 15.6 30.6
A B POD FAR CSI a=71 b=1
99.52 4.50 6.25 98.61 1.15 c=18018 d=15
September Summary
![Page 18: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/18.jpg)
Daily Summaries
![Page 19: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/19.jpg)
Synoptic Scale Meteorology 4 August
.01 0.5 1.0 2.0 3.0
20:15
![Page 20: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/20.jpg)
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
44.1
(58.5)
55.8
(62.1)
0.41
(0.27)
11.7
(3.5)
26.5
(6.0)
17.9
(13.1)
32.8
(17.3)
48 Hour Max. 8-Hour Ozone Forecast Init: 3 Aug (12 Z) Valid: 4 Aug TP (04 Z to 03 Z)
Model performance is poorer alongthe cold front where cloud cover andheavy rain dominate
![Page 21: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/21.jpg)
Synoptic Scale Meteorology 5 August
.01 0.5 1.0 2.0 3.0
20:15
![Page 22: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/22.jpg)
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
43.8
(58.2)
51.6
(63.0)
0.73
(0.66)
7.8
(4.9)
17.9
(8.4)
12.3
(10.4)
22.3
(14.5)
48 Hour Max. 8-Hour Ozone Forecast Init: 4 Aug (12 Z) Valid: 5 Aug TP (04 Z to 03 Z)
Model performance is still poor alongthe cold front where cloud cover andheavy rain dominate
![Page 23: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/23.jpg)
Synoptic Scale Meteorology 6 August
.01 0.5 1.0 2.0 3.0
20:15
![Page 24: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/24.jpg)
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
37.8
(52.6)
44.3
(55.3)
0.82
(0.67)
6.4
(2.6)
17.0
(5.0)
8.7
(6.9)
18.9
(10.1)
48 Hour Max. 8-Hour Ozone Forecast Init: 5 Aug (12 Z) Valid: 6 Aug TP (04 Z to 03 Z)
Model performance is very good throughoutthe domain, except for sections of NewEngland and southeast Ontario, where cloudcover is greatest.
![Page 25: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/25.jpg)
Synoptic Scale Meteorology 7 August
.01 0.5 1.0 2.0 3.0
20:15
![Page 26: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/26.jpg)
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
41.2
(50.1)
45.9
(50.7)
0.79
(0.71)
4.7
(0.6)
11.5
(1.3)
8.6
(5.3)
16.2
(8.2)
48 Hour Max. 8-Hour Ozone Forecast Init: 6 Aug (12 Z) Valid: 7 Aug TP (04 Z to 03 Z)
Model performance is again very good, except for sections of New England, southeastOntario and Wisconsin, where cloud cover is greatest.
![Page 27: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/27.jpg)
Synoptic Scale Meteorology 8 August
.01 0.5 1.0 2.0 3.0
20:15
![Page 28: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/28.jpg)
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
48.7
(54.8)
52.0
(55.2)
0.75
(0.62)
3.3
(0.4)
6.8
(0.7)
9.6
(6.6)
14.5
(9.5)
48 Hour Max. 8-Hour Ozone Forecast Init: 7 Aug (12 Z) Valid: 8 Aug TP (04 Z to 03 Z)
Model continues to perform well as thecP airmass undergoes modification. Warmertemperatures allow for higher ozone values (mean observed up to 48.7). New Englandstill has more cloud cover.
![Page 29: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/29.jpg)
Synoptic Scale Meteorology 9 August
.01 0.5 1.0 2.0 3.0
20:15
![Page 30: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/30.jpg)
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
52.9
(57.1)
55.2
()58.1
0.78
(0.61)
2.3
(1.0)
4.4
(1.8)
8.0
(7.4)
11.6
(9.8)
48 Hour Max. 8-Hour Ozone Forecast Init: 8 Aug (12 Z) Valid: 9 Aug TP (04 Z to 03 Z)
Good model performance continues, as thecP airmass continues to warm. Highestozone values (52.9) observed on this day. Cloud cover in New England dissipates resulting in better model performance there.
![Page 31: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/31.jpg)
Synoptic Scale Meteorology 10 August
.01 0.5 1.0 2.0 3.0
20:15
![Page 32: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/32.jpg)
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
49.4
(58.4)
57.7
(60.8)
0.69
(0.59)
8.6
(2.4)
17.4
(4.2)
15.6
(9.4)
24.5
(12.6)
48 Hour Max. 8-Hour Ozone Forecast Init: 9 Aug (12 Z) Valid: 10 Aug TP (04 Z to 03 Z)
Model performance degrades somewhat as cloud cover and precipitation increases overthe domain, most notably with a cold frontapproaching from the Great Lakes and a low forming over south
![Page 33: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/33.jpg)
Synoptic Scale Meteorology 11 August
.01 0.5 1.0 2.0 3.0
20:15
![Page 34: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/34.jpg)
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
42.2
(51.8)
54.9
(60.7)
0.71
(0.33)
12.7
(9.0)
30.2
(17.3)
15.7
(12.7)
31.7
(19.3)
48 Hour Max. 8-Hour Ozone Forecast Init: 10 Aug (12 Z) Valid: 11 Aug TP (04 Z to 03 Z)
Model performance continues to suffer over the Great Lakes area as cloud cover, precipitation and very cool temperatures dominate this region.
![Page 35: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/35.jpg)
Synoptic Scale Meteorology 12 August
.01 0.5 1.0 2.0 3.0
20:15
![Page 36: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/36.jpg)
Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
32.5
(44.9)
51.1
(56.6)
0.38
(0.18)
18.6
(11.7)
57.4
(26.1)
21.0
(14.1)
58.0
(28.1)
48 Hour Max. 8-Hour Ozone Forecast Init: 11 Aug (12 Z) Valid: 12 Aug TP (04 Z to 03 Z)
Model performance at its worst as extensive cloud cover, precipitation and very cool temperatures dominate the entire domain.
![Page 37: A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS Workshop Chapel Hill, NC 20 October, 2004](https://reader035.vdocuments.net/reader035/viewer/2022062712/56649c925503460f9494cd8f/html5/thumbnails/37.jpg)
Monthly Summary Obs
Mean
Model
Meanr
MB
(ppb)
NMB
(%)
RMSE
(ppb)
NME
(%)
June 46.1 53.9 0.51 7.8 16.9 14.1 24.0
July 47.4 57.3 0.55 9.9 20.8 16.2 27.2
August 43.4 55.1 0.62 11.7 27.1 16.5 30.9
September 41.7 52.8 0.65 11.1 26.7 15.6 30.6
Summary
Model performed reasonably well over the four month period, despiteanomalously cool and wet conditions.
Performance was closely tied to meteorology:
M very good performance when skies were generallyclear and void of precipitation,
M poorer when cloud cover and precipitation occurred.