will canadian precipitation analysis system improve ... canadian precipitation analysis system...

1

Will Canadian Precipitation Analysis System improve precipitation estimates in Alberta?

1 University of Alberta, Western Partnership of Wildland Fire Science 2 Natural Resources Canada, Canadian Forest Service

Xinli Cai1, Piyush Jain1, Xianli Wang2 , Mike Flannigan1

Today I will present my thesis project: which is comparing CaPA system with interpolation methods in estimating precipitation in Alberta.

2

1Development and structure of the Canadian Forest Fire Weather Index System. 1987. Van Wagner, C.E. Canadian Forestry Service, Headquarters, Ottawa. Forestry Technical Report 35. 35 p.

Canadian Fire Weather Indexes (FWI) System1

Precipitation >0.5mm



As you all may be familiar with the FWI system is the fundamental unit of fire danger rating for fire management agencies in Canada. The FWI system depends on the daily observations of temperature, relative humidity, wind speed, and 24h precipitation. As you can see, precipitation directly affects the fuel moisture codes, which are FFMC, DMC, and DC These moisture codes are impacted at different precipitation thresholds, which are listed here

3

The highly variable nature of spatial distribution of precipitation (PCP) is the biggest challenge for the spatial interpolation of FWI indexes2,3.

2Flannigan, M. D., and Wotton, B. M., (1989) A study of interpolation methods for forest fire danger rating in Canada. Can. J. For.Res. 19: 1059-1066.3Flannigan, M. D., Wotton, B. M., and Ziga, S., (1998) A study on the interpolation of fire danger using radar precipitation estimates. International Journal of Wildland Fire. 8:217-225.

Challenges

Estimating precipitation is currently one of the key challenges of accurate fire danger rating. As you can see in the radar image, the occurrence and amount of observed precipitation is highly variable. However, provincial and federal agencies do not use any radar observations. They use a simple interpolation algorithm called inverse distance weighting, impact of the surrounding observations decrease with distance. As you can see in the idw product, there is no fine detail.

4



Challenges

PRECIPT – Rain – 2015 – 07 – 18, 12:50 UTC, 1/13

Radar observed PCP on July 18, 2015


5



Challenges

PRECIPT – Rain – 2015 – 07 – 18, 12:50 UTC, 1/13

Radar observed PCP on July 18, 2015

Interpolated PCP on July 18,2015 using inverse distance weighting (IDW)


6

(produced by Environment and Climate Change Canada)

Potential Solutions 1: Canadian Precipitation Analysis (CaPA) System4

CaPA Radar

Station observations

GEM forecasts

4Mahfouf et al. (2007). A Canadian precipitation analysis (CaPA) project: Description and preliminary results. AtmosphereOcean,45:1-17.

• Use optimal interpolation procedure to combine the three sources

• Produce near real time 10km gridded PCP

One potential solution is the CaPA system, which is a product from Environment and Climate Change Canada. The CaPA system integrates station observations, radar and GEM forecasts (Global Environment Multiscale) at 10 km gridded resolution This map shows the radar coverage, which extends 120km beyond the radar station location. As well, ECCC weather stations are shown in the dark points. As you can see, the radar coverage and wx location are mostly in the agricultural areas

7

(produced by Environment and Climate Change Canada)

Potential Solutions 1: Canadian Precipitation Analysis (CaPA) System4

CaPA Radar

Station observations

GEM forecasts

4Mahfouf et al. (2007). A Canadian precipitation analysis (CaPA) project: Description and preliminary results. AtmosphereOcean,45:1-17.

• Use optimal interpolation procedure to combine the three sources

• Produce near real time 10km gridded PCP

One potential solution is the CaPA system, which is a product from Environment and Climate Change Canada. The CaPA system integrates station observations, radar and GEM forecasts (Global Environment Multiscale) at 10 km gridded resolution This map shows the radar coverage, which extends 120km beyond the radar station location. As well, ECCC weather stations are shown in the dark points. As you can see, the radar coverage and wx location are mostly in the agricultural areas

8

Potential Solution 2: use other interpolation methods

The second potential solution is using more advanced interpolation methods. Like thin-plate splines interpolation. The principle of TPS is to fit a field with minimum mean-square error and a smooth curvature constraint. Another advanced interpolation method could be used is ordinary kriging. Ordinary kriging models the spatial variability using variogram modeling. Interpolation: Estimation of a variable at an unmeasured location from observed values at surrounding locations.

9

5Cressie, N., (1993) Statistics for Spatial Data. John Wiley and Sons, 900 pp.

Thin Plate Spline5 (TPS): smooth and non-smooth

A Thin Plate Splines surface


A field is fitted with minimum mean-square error under a smooth curvature constraint.


10

5Cressie, N., (1993) Statistics for Spatial Data. John Wiley and Sons, 900 pp.

Thin Plate Spline5 (TPS): smooth and non-smooth

A Thin Plate Splines surface

Ordinary Kriging5 (ok)

Exponential Model


A geostatistical method that models spatial variability using regression analysis on the covariance structure as a function of distance (i.e., variogram modeling).

A field is fitted with minimum mean-square error under a smooth curvature constraint.


11

Potential Solution 3: A combination of solution 1 and solution 2

Regression Kriging with CaPA (rk_capa)

• Fit a regression model using observed precipitation and CaPA outputs• Then, build the variogram model using the residuals

The third potential solution is a combination of the first and second solutions. Specifically, the method is called regression kriging with CaPA. The method is similar to ordinary kriging but use CaPA as an auxiliary variable. So the variogram shown here is built using the residuals of observed PCP and CaPA outputs.

12

1. Is CaPA superior in estimating PCP in Alberta? • What about radar versus non-radar areas?

Research Questions

2. What are the implications of improved PCP estimates on FWI indexes?

3. How sensitive is the PCP and FWI to weather station density?

There are three research question we wish to answer:

13

Leave-one-out cross-validation (LOOCV)

Methodology

To answer the questions, we used a leave one out cross validation procedure to generate precipitation estimates in the interior regions of Alberta The LOOCV procedure removes one weather station at a time and uses the rest of the weather stations to estimate the PCP at that location. Next, the weather station is replaced and another weather station is removed. This removal continues until all weather stations have been removed and replaced once. In this study, we compared the following 10 methods. Some methods use a square root transformation and in the last column we provide the acronyms.

14


Data: Alberta fire weather station observations for 2014 (14th July – 31st August) and 2015 (1st May to 31st August). Source: Alberta Agriculture and Forestry (AAF)

Methodology

Areas are divided into validated areas to remove the edge effect


15


Candidate MethodsTransformation

of precipitationAcronym

Canadian precipitation analysis system – CaPA

Inverse distance weighting – idw

Thin-plate spline non-smoothing – tps_ns

Thin-plate spline non-smoothing square root tps_ns_s

Thin-plate spline smoothing – tps_s

Thin-plate spline smoothing square root tps_s_s

Ordinary kriging – ok

Ordinary kriging square root ok_s

Regression kriging with CaPA – rk(capa)

Regression kriging with CaPA square root rk(capa)_s

Data: Alberta fire weather station observations for 2014 (14th July – 31st August) and 2015 (1st May to 31st August). Source: Alberta Agriculture and Forestry (AAF)

Methodology

Areas are divided into validated areas to remove the edge effect


16

Results 1.1: Performance rank of the 10 PCP prediction methods

�� (��) = �� × ��=�� =�� ,� − ��,�n= number of weather stations (i.e.81); k= number of days (i.e.123)

Best

Worst

First, we compared the 10 methods using the Mean Absolute error. The MAE measures the magnitude of the errors. A smaller MAE means the methods was more accurate. As you can see, in this 95% CI plot of MAE, CaPA had a mid-tiered performance and was 6-7% better than IDW ok_sqrt, rk_capa_sqrt, tps_s_sqrt were the top three methods and were 23%, 21%, and 19% better than IDW. Transformation of PCP in general greatly improved the accuracy. Bias measures the direction of the errors. The red line represents no bias. Here we can see CaPA had a large tendency to underestimate PCP. Also, we can see that methods using square root transformation underestimate PCP.

17

Results 1.1: Performance rank of the 10 PCP prediction methods

�� (��) = �� × ��=�� =�� ,� − ��,� Mean error (B��) = ��×�∑�=�� ∑�=�� (��,� −��,�)n= number of weather stations (i.e.81); k= number of days (i.e.123)

Best

Worst

First, we compared the 10 methods using the Mean Absolute error. The MAE measures the magnitude of the errors. A smaller MAE means the methods was more accurate. As you can see, in this 95% CI plot of MAE, CaPA had a mid-tiered performance and was 6-7% better than IDW ok_sqrt, rk_capa_sqrt, tps_s_sqrt were the top three methods and were 23%, 21%, and 19% better than IDW. Transformation of PCP in general greatly improved the accuracy. Bias measures the direction of the errors. The red line represents no bias. Here we can see CaPA had a large tendency to underestimate PCP. Also, we can see that methods using square root transformation underestimate PCP.

18

• A non-parametric resampling hypothesis test6 was applied for the spatial and temporal autocorrelation of the data. • Stationary block bootstrapping7 (SBB) was implemented (n=1000 times). • Resampling ANOVA test followed by resampling Post-hoc paired t-tests with Holm-Bonferroni p-value adjustment

6Politis &Romano. (1994). The Stationary Bootstrap. Journal of the American Statistical Association, Vol.89:1303-1313.7Hamill, T. (1999). Hypotehsis test for evaluating numerical precipitation forecasts. America Meteorological Society, 155-167.

Test statistic:

MAE(capa) – MAE(idw)

5 % CI 95 % CI

Results 1.2: Statistical tests of the PCP prediction methods

To see if the methods were statistically different, a non-parametric resampling hypothesis test was applied. This procedure implemented the stationary block bootstrapping to a resampling ANOVA test followed by resampling Post-hoc paired t-test. I won’t get into the math today, but the details can be find in my thesis! Paired comparisons are shown in this table. The top rows are the MAEs of the 10 methods and listed from the worst to the best. Here, a circle represents stat. sig. in 2014 and triangle represents stat. sig. in 2015 As you can see, the top three methods had statistically lower MAE than most of the PCP prediction methods in 2015 but there was no significance between them.

19

• A non-parametric resampling hypothesis test6 was applied for the spatial and temporal autocorrelation of the data. • Stationary block bootstrapping7 (SBB) was implemented (n=1000 times). • Resampling ANOVA test followed by resampling Post-hoc paired t-tests with Holm-Bonferroni p-value adjustment

6Politis &Romano. (1994). The Stationary Bootstrap. Journal of the American Statistical Association, Vol.89:1303-1313.7Hamill, T. (1999). Hypotehsis test for evaluating numerical precipitation forecasts. America Meteorological Society, 155-167.

Test statistic:

MAE(capa) – MAE(idw)

5 % CI 95 % CI

Overall MAE (mm) for 2014 and 2015 (test statistics)

Method ok idw

tps

_ns tps _s

rk

(capa) CaPA

tps

_ns_s

tps

_s_s

rk

(capa)_s ok_s

2014 1.47 1.42 1.37 1.33 1.34 1.33 1.25 1.14 1.16 1.12

2015 1.31 1.30 1.27 1.24 1.23 1.20 1.16 1.08 1.03 1.02

ok – ▲ ▲ ▲idw – ▲ ●▲ ▲ ▲

tps_ns – ▲ ▲ ▲ ▲tps _s – ▲ ▲ ●▲ ▲

rk(capa) – ▲ ▲ ▲CaPA – ▲ ▲ ▲

tps_ns_s – ▲ ▲ ▲tps_s_s –

rk(capa)_s –

ok_s –

● : significant in year 2014

▲: significant in year 2015

Results 1.2: Statistical tests of the PCP prediction methods

To see if the methods were statistically different, a non-parametric resampling hypothesis test was applied. This procedure implemented the stationary block bootstrapping to a resampling ANOVA test followed by resampling Post-hoc paired t-test. I won’t get into the math today, but the details can be find in my thesis! Paired comparisons are shown in this table. The top rows are the MAEs of the 10 methods and listed from the worst to the best. Here, a circle represents stat. sig. in 2014 and triangle represents stat. sig. in 2015 As you can see, the top three methods had statistically lower MAE than most of the PCP prediction methods in 2015 but there was no significance between them.

20

Spatial distribution of MAE in 2015

Results 1.3: Spatial variation between the PCP prediction methods

The MAEs are shown spatially by weather station. A dark dot means poorer performance. It is very hard to address the spatial variability of these methods, but it is clear to show in this figure the spatial variability exit.

21

MAE Performance Ranking: Radar versus Non-radar

• CaPA, rk(capa), and rk(capa)_s were the most improved methods in radar covered regions.

IDW

rk(capa)_sqrt

ok_sqrt

tps_ns_sqrt

tps_s_sqrt

CaPA

ok

tps_ns

tps_s

rk(capa)

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10

Per

form

ane

ran

k o

f P

CP

can

did

ate

met

ho

ds

in r

adar

are

a

Performance rank of PCP candidate methdos in non-radar area


In this slide, we show the MAE performance rank in radar and non-radar regions. The best performing method had a rank of 1, while the worst performing method had a rank of 10. Methods under the 1 to 1 line had improved performance under radar coverage. For example, CaPA ranked the 5th in non-radar area and rankd the 2nd in radar covered area. As you can see, CaPA related methods greatly improved under radar and rk(capa)_sqrt was even better than CaPA

22

MAE Performance Ranking: Radar versus Non-radar

• CaPA, rk(capa), and rk(capa)_s were the most improved methods in radar covered regions.

IDW

rk(capa)_sqrt

ok_sqrt

tps_ns_sqrt

tps_s_sqrt

CaPA

ok

tps_ns

tps_s

rk(capa)

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10

Per

form

ane

ran

k o

f P

CP

can

did

ate

met

ho

ds

in r

adar

are

a

Performance rank of PCP candidate methdos in non-radar area


In this slide, we show the MAE performance rank in radar and non-radar regions. The best performing method had a rank of 1, while the worst performing method had a rank of 10. Methods under the 1 to 1 line had improved performance under radar coverage. For example, CaPA ranked the 5th in non-radar area and rankd the 2nd in radar covered area. As you can see, CaPA related methods greatly improved under radar and rk(capa)_sqrt was even better than CaPA

23

Frequency Bias Index(FBI)=(H+F)/(H+M)-1

Measures the frequency of forecast events, best score is 0.

Equitable Threat Score (ETS) = H-HR / (H+M+F-HR)

Where HR =�+� (�+�)�

Measures the accuracy of forecast events, best score is 1.

Results 1.4: PCP prediction methods for fuel moisture codes

PCP category: 0-0.5mm, 0.5-1.5mm, 1.5-2.5mm, >2.8mm

Daily precipitation is highly skew towards zero, where more than half of the days received no rain. Therefore, we evaluated the methods based on the minimal precipitation thresholds of FFMC, DMC, and DC using common meteorological metrics: equitable threat score and frequency bias index. ETS uses the contingency table (list here) to measure PCP prediction accuracy and uses an adjusted factor such that completely random forecasts produce a score of zero (i.e., no skill). FBI uses the same contingency table to measures whether a method tends to overestimate or underestimate PCP. The ETS barplot confirmed that ok_sqrt, rk(capa)_sqrt, and tps_s_sqrt are the best methods in all PCP categories The FBI barplot show that from 0.5 to 2.8mm, all methods over-estimate PCP. ZOOM IN: CaPA (light blue) had the highest tendency to underestimate PCP>2.8mm. We will examine the impact of CaPA’s tendency of underestimate big PCP event in the next slide. Note: Our observation ofCaPA greatly underestimated large PCP events agrees with Fortin, V. R. (2015)

24




Where HR =�+� (�+�)�





25




Where HR =�+� (�+�)�





26

MAE of FWI indexes calculated with estimated PCP and observed RH, WS, Temp for 2015 (1st May to 31st August).

BestWorst

Results 2.1:Implications to FWI indexes

Methods PCP FFMC ISI FWI

capa 1.2 4.23 0.69 2.74

idw 1.3 4.58 0.82 2.8

tps_ns 1.27 4.2 0.71 2.52

tps_ns_sqrt 1.16 3.92 0.64 2.41

tps_s 1.24 4.29 0.76 2.64

tps_s_sqrt 1.08 3.79 0.63 2.47

rk(capa) 1.23 4.06 0.75 2.61

rk(capa)_sqrt 1.03 3.57 0.58 2.36

ok 1.31 4.65 0.9 3.05

ok_sqrt 1.02 3.62 0.59 2.46

We assessed the implications of improved PCP estimates to FWI indexes. We divided the FWI indexes into two groups according to their drying rate. We first show the FFMC, ISI, and FWI which have quicker drying rates and respond quickly to PCP events. This figure shows that Rk(capa)_sqrt was consistently the best method for FFMC, ISI, and FWI. On the other hand, DC, DMC, and BUI have longer drying rates and affected only by big PCP events were presented in the right figure. As you can see, CaPA, which had the largest underestimation of big PCP events resulted in the worst performance for DMC, DC, and BUI. Also, Rk(capa) was the best method for DC, DMC, and BUI. Therefore, rk_capa_sqrt (method have the highest accuracy) was performed the best for quick drying indexes (FFMC, ISI, and FWI). Rk_capa (method have the least tendency to underestimate big PCP events ) was performed the best for slow drying indexes (DMC, DC, and BUI).

27

Methods PCP FFMC ISI FWI DMC DC BUI

capa 1.2 4.23 0.69 2.74 9.68 66.52 13.49

idw 1.3 4.58 0.82 2.8 7.98 49.78 10.74

tps_ns 1.27 4.2 0.71 2.52 7.79 49.95 10.76

tps_ns_sqrt 1.16 3.92 0.64 2.41 7.82 49.27 10.8

tps_s 1.24 4.29 0.76 2.64 7.98 46.99 10.86

tps_s_sqrt 1.08 3.79 0.63 2.47 8.68 53.75 11.89

rk(capa) 1.23 4.06 0.75 2.61 7.75 45.54 10.65

rk(capa)_sqrt 1.03 3.57 0.58 2.36 8.48 55.22 11.76

ok 1.31 4.65 0.9 3.05 8.46 48.31 11.38

ok_sqrt 1.02 3.62 0.59 2.46 9.22 60.24 12.76

MAE of FWI indexes calculated with estimated PCP and observed RH, WS, Temp for 2015 (1st May to 31st August).

BestWorst

FBI >2.8mm

Remember, CaPA had high negative bias for PCP > 2.8mm

Results 2.1:Implications to FWI indexes

Methods PCP FFMC ISI FWI

capa 1.2 4.23 0.69 2.74

idw 1.3 4.58 0.82 2.8

tps_ns 1.27 4.2 0.71 2.52

tps_ns_sqrt 1.16 3.92 0.64 2.41

tps_s 1.24 4.29 0.76 2.64

tps_s_sqrt 1.08 3.79 0.63 2.47

rk(capa) 1.23 4.06 0.75 2.61

rk(capa)_sqrt 1.03 3.57 0.58 2.36

ok 1.31 4.65 0.9 3.05

ok_sqrt 1.02 3.62 0.59 2.46

We assessed the implications of improved PCP estimates to FWI indexes. We divided the FWI indexes into two groups according to their drying rate. We first show the FFMC, ISI, and FWI which have quicker drying rates and respond quickly to PCP events. This figure shows that Rk(capa)_sqrt was consistently the best method for FFMC, ISI, and FWI. On the other hand, DC, DMC, and BUI have longer drying rates and affected only by big PCP events were presented in the right figure. As you can see, CaPA, which had the largest underestimation of big PCP events resulted in the worst performance for DMC, DC, and BUI. Also, Rk(capa) was the best method for DC, DMC, and BUI. Therefore, rk_capa_sqrt (method have the highest accuracy) was performed the best for quick drying indexes (FFMC, ISI, and FWI). Rk_capa (method have the least tendency to underestimate big PCP events ) was performed the best for slow drying indexes (DMC, DC, and BUI).

28

Weather station density = 2.69 wstn per 10,000 km2

Note: The ECCC weather station density of CaPA was constant in the study area (~1.58 wstn per 10,000 km2 in 2015)

Results 3: the sensitivity of PCP/ FWI estimates to fire weather station density

ScenarioNo. of

stations

No. of stations per

10000km2

10% selected 14 0.28



75% selected 102 2.02

90% selected 123 2.43

Weather station density by randomly selecting weather stations

from the study area (rounded up)

Additionally, we examined the sensitivity of PCP and FWI estimates to fire weather station density. Briefly, the sensitivity analysis was repeated the aforementioned analysis (LOOCV) 100 times by randomly selecting 10%, 25%, 50%, 75%, and 90% of the weather station in the study area. The total weather station density in our study area was 2.69 wstn per 10,00 km2 in 2015 and the corresponded monk weather station for sensitivity analysis is shown in the table. The plot of PCP MAEs of the 10 methods versus the weather station density shows that all the interpolation methods had improved performance with higher weather station density. CaPA (light blue) is superior than the rk(capa)_sqrt in the scenario of weather station density below ~0.6wstn / 10,000km2, superior than the ok_sqrt of weather station density below ~1 wstn / 10,000km2, and superior than the tps_s_sqrt of weather station density below ~1.3 wstn / 10,000km2. The similar results were observed for FWI index as well.

29




ScenarioNo. of

stations

No. of stations per

10000km2




75% selected 102 2.02

90% selected 123 2.43




30




ScenarioNo. of

stations

No. of stations per

10000km2




75% selected 102 2.02

90% selected 123 2.43




31

Conclusions and implications

1. CaPA had a mid-tiered performance amongst the 10 candidate methods and was significantly better than IDW (6%-7% improvement). However, CaPA had a large tendency to underestimate PCP>2.8mm.

2. ok_sqrt, rk_capa_sqrt, and tps_s_sqrt were the top three methods and were significantly better than other methods (i.e., 23%, 21%, and 19% improvement compared to IDW).

3. CaPA related methods greatly improved under radar and rk_capa_sqrt was even better than CaPA.

4. Improved PCP estimates significantly improved FWI indexes. rk_capa_sqrt was the top method for quick drying indexes: FFMC, ISI, and FWI, while rk_capa was the top method for slow drying indexes: DC, DMC, and BUI.

5. CaPA is better than interpolation methods when there are <0.6 wstns per 10 000km2.

CaPA is a great “seed” value in regression kriging but should not be used directly (especially if wstn density >0.6 stns/10 000km2)

32

Take home PCP / FWI decision chart

Is the area under radar?

Is the density > 0.6 wstn / 10 000 km2?

yes

rk(CaPA)_sqrt

yes no

CaPArk(CaPA)_sqrt

no

33

Thank you, any questions?

34

Sensitivity of DMC/DC estimates to fire weather station density

Appendix

Use rk(CaPA) regardless of wstn density!

Appendix will be needed if someone asked related questions

35

idw

ok

ok_ln

ok_c

ok_s

rk(capa)_ln

rk(capa)_c

rk(capa)_s

rk(capa)

tps_s_ln

tps_s_c

tps_s_s

tps_s

tps_ns_ln

tps_ns_c

tps_ns

CaPA

tps_ns_s

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

2015 P

erfo

rm

an

ce r

an

k o

f can

did

ate

meth

od

s

2014 Performance rank of candidate methods

MAE

Appendix

Performance ranking of 18 PCP prediction methods between year 2014 and 2015

Performance rank of the 18 PCP candidate methods were similar (stable) between 2014 and 2015.

The best performing method (lowest

MAE) had a rank of 1, while the worst

performing method (biggest MAE) had

a rank of 18.

36

Appendix

Histogram of monthly precipitation (mm/month) for all the active weather stations for 2014, 2015, and 30-year average (1984 –

2013) in the overall study area.

Both 2014 and 2015 were drier than the 30-yrs average and 2014 was especially dry.

37

Appendix

Averaged temporal autocorrelation indicate the mean block length (m) that requires for

the Stationary Block Bootstrapping (see section 2.3.2.2a) to remove the temporal

correlation of PCP and FWI indexes in the resampling hypothesis test.

In this study, we chose m of 3 days for PCP; m of 4 days for FFMC; m of 5 days for

FWI; m of 12 for DMC.

Averaged temporal autocorrelation of daily PCP and FWI indexes for fire weather stations (n=81) in 2015.

will canadian precipitation analysis system improve ... canadian precipitation analysis system...

Documents