detection of inhomogeneities in daily climate records to study trends in extreme weather detection...

Detection of inhomogeneities in Daily climate records to Study Trends in Extreme Weather

Detection of Breaks in Random Data,

in Data Containing True Breaks, and in Real Data

Ralf Lindau

Daily Stew Meeting, Bonn – 14. June 2012

Internal and External Variance

Consider the differences of one station compared to a neighbour or a reference.

Breaks are defined by abrupt changes in the station-reference time series.

Internal variancewithin the subperiods

External variancebetween the means of different

subperiods

Criterion:Maximum external variance attained bya minimum number of breaks


Decomposition of Variance

n total number of yearsN subperiodsni years within a subperiod

The sum of external and internal variance is constant.


Three Questions

How do random data behave?

Needed as a stop criterion for the number of significant breaks.

How do real breaks behave theoretically?

How do real data behave?


Segment averages

with stddev = 1

Segment averages xi scatter randomly

mean : 0

stddev: 1/

Because any deviation from zero can beseen as inaccuracy due to the limited number of members.

in


2-distribution

The external varianceis equal to the mean square sumof a random standard normal distributed variable.

Weighted measure for thevariability of the subperiods‘means


From 2 to distribution

n = 21 yearsk = 7 breaks

data

X ~ 2(a) and Y ~ 2(b)

X / (X+Y) ~ (a/2, b/2)

If we normalize a chi2-distributed variable by the sum of itself and another chi2-distributed variable, the result will be -distributed.

)(

)()(),(

ba

babaB

2

1,2

1)(

12

112

knkB

vvvp

knk

with


Incomplete Beta Function

2

1,2

1)(

12

112

knkB

vvvp

knk

External variance v is -distributedand depends on n (years) and k (breaks):

2

ki

1

0

1)(i

l

lml vvl

mvP

Solvable for even k and odd n:

2

3n

m

The exceeding probability P gives thebest (maximum) solution for v

Incomplete Beta Function

v

pdvvP0

1)(

We are interested in the best solution, with the highest external variance.We need the exceeding probability for high varext


P(v) for different k

Can we give a formula for

in order to derive v(k)?

220

breaksdk

dv

Increasing the break number from k to k+1 has two consequences:

1. The probability function changes.

2. The number combinations increase.


dv/dk sketch

P(v) is a complicated function and hard to invert into v(P).

Thus, dv is concluded from dP / slope.

And the solution is:

k breaks

k+1 breaks

1

0

1)(i

l

lml vvl

mvP

vk

vkn

k

knc

kn

v

dk

dv

1

1ln2

11ln

1

12


Solution

5ln21ln

2

1

1

1*

***

k

kk

dk

dv

v

k

***

*

1

5ln21ln

2

1

1

1dk

kk

kdv

v

*

2

1

*

*

2

1)5ln(2* 1

11k

k

kkv


Constance of Solution

101 ye

ars21 yea

rs

The solution for the exponent is constant for different length oftime series (21 and 101 years).


The extisting algorithm Prodige

Original formulation of Caussinus and Mestre for the penalty term in Prodige

Translation into terms used by us.

Normalisation by k* = k / (n -1)

Derivation to get the minimum

In Prodige it is postulated that the relative gain of external variance is a constant for given n.

minln21ln * nkv

0ln21

1*

ndk

dv

v

ndk

dv

vln2

1

1*

minln1

21ln

n

n

kv

min)ln(

1

2

)(

)(

1ln)(

1

2

1

1

2

nn

lk

YY

YYn

YCn

ii

k

j

jj

k


Our Results vs Prodige

We know the function for the relative gain of external variance.

Its uncertainty as given by isolines of exceeding probabilities for 2-i are characterised by constant distances.

Prodige propose a constant of 2 ln(n) ≈ 9

Exceeding probability1/1281/641/321/161/81/4


Wrong Direction

n = 101 years n = 21 years

True Breaks


Only true for constant lengths

True breaks with fixed distances behave identical to random data.

For realistic random lengths the exponent is slightly increased.


Sub-periods withrandom lengths

Sub-periods withconstant lengths

data

theory

theory

data

Distribution of Lengths

The distribution of the sub-periods’ lengths as obtained by randomly inserted breaks is known.

If necessary, it could be taken into account.


Break vs Scatter Regime

The two governing parameters are:

1) The relative amount of break variance compared to the scatter variance

2) The quotient

The latter defines how much faster the internal variance decreases in the “true break regime” compared to the “scatter regime”

If the relative scatter is low (10%) the transition between the regimes is clearly visible at 15 from 19 breaks.


Time series lengthNumber of true breaks

Real Data

1050 Climate Stations exist in Germany.

For each station the next eastward (to avoid identical pairs) neighbour between 10 km and 30 km is searched.

443 stations pairs remain.


All Stations Neighbouring pairs

Data Focus

This project deals with daily climate data.

Findings about their extremes are in the focus.

At least statements about the

•distribution (moments)

•percentiles

•indices (number of wet days per month)

should be possible.


Parameters


Interesting for break detection:

Problem parameters PP

Expected physical problems

Temperature at high sun shine duration

Temperature at high pressure

Temperature at high diurnal cycle

Temperature during snow cover

Temperature depending on general

weather situation

Temperature during rain

Rain at high wind speed

Expected technical problems

Frequency of rainy days below 1 mm

Tenth of precipitation report

Difference between Tmean and (Tmax-Tmin)

Per se interesting parameters P

Monthly means

Temperature

Precipitation, etc.

Breaks are moresensitive to problem parameters. Breaks in PP may help to find breaks in P

Distribution and extremes

Standard deviationSkewnessKurtosisMaximumMinimum90 percentile

project focus(more sensitive?)

Two Parameter Pairs

1a. Monthly mean temperature

1b. Monthly maximum temperature

2a. Monthly precipitation sum

2b. Frequency of rainy days below 1 mm

Can the sensitive parameter help to find breaks in the mean?


(Project focus)

(Problem parameter)

“Drizzle days” are often excluded from rainy days to calculate the interesting indices:

•Monthly Rain Frequency •Consecutive Dry Days

“Drizzle frequency” is not only a technical problem parameter, but also a per se interesting one.

Monthly Mean Temperature


Temperature differencebetween Ellwangen-RindelbachandCrailsheim-Alexandersreutshows 1 strong and 3 further significant breaks.

The statistical signature confirms it:The first break contains much variance.2, 3 and 4 are only slightly larger than the Mestre penalty.

Break Statistics


Individual pair All pairs

r = 0.937

Monthly Maximum

For the monthly temperature maximum, only the largest breaks are detectable, probably due to the reduced correlation.


r = 0.865

Additional Breaks?In maximum temperature there are less breaks. Are they nevertheless new compared to those in mean temperature?

Enhance the penalty from about 12 (i.e. 2 ln(n)) to 60.)

With n = 600, it means that 10% of the remaining internal variance has to be explained by each additional break. Otherwise the search is stopped.

For such increased requirements 297 breaks are found in the mean and 67 in the maximum.

Nearly all breaks in tmax exist also in tmean.

The “stddev” of temporal distance is 1.75 years.


Answer: No

Nearly no new break is found by the sensitive parameter Monthly Maximum Temperature.

The lower correlation (0.865 vs. 0.937 doubled rms) hamper obviously the break finding capability of the sensitive parameter.

However, the high correlation of break positions may the opposite direction become possible: To find break positions in the maximum temperature by considering the mean temperature.


“Drizzle Days”

Monthly frequency of rainy days below 1mm.

This parameter is highly inhomogeneous.

Even for individual stations the break is evident.


Drizzle vs. Mean Precip.


In the drizzle parameter moresignificant breaks are found (index 43.3 compared to 28.8),although the correlation is low,(0.339 compared to 0.855).

Are the break positions againcorrelated?

Correlation of break positions

Many new breaks are found. Only 12 breaks of the drizzle parameter are found at all somewhere the corresponding time series of mean precipitation, but mostly far away.

In 93 time series pairs one or more breaks are found for drizzle, but even not a single in mean precipitation.

Are these new breaks also included, but hidden in mean precipitation?


remember

Forced Breaks (1)


Forced Breaks (2)

Also in average, the external variance decreases only by about 1%, if “drizzle breaks” are inserted into the time series of mean precipitation.

1% is the mean decrease of a random n=100 time series and it is beta-distributed.

However, here n is equal to 600. Is the result then a bit better than random?


Simulated Data

1. Blind try of 3 breaks in a 21 years random time series

2. Blind try of 3 breaks in a 21 years constant time series with 6 true breaks.

3. Blind try 3 breaks in a 21 years time series with 6 true breaks plus random scatter.


1. Purely random 2. Pure true breaks 3. Realistic mix

Realistic Mixed Data

Real data is expected to be similar to a realistic mix, rather than to random scatter.

As it then includes also real breaks, the Null Hypothesis is not random scatter, but a realistic mix.

Here the blindly found external variance is again -distributed, but generally larger. How much is difficult to quantify in advance . It depends on the signal to noise ratio.



Conclusions• The analysis of random data shows that the external variance is -distributed,

which leads to a new formulation for the penalty term.

• True breaks are also -distributed. Their external variance increases faster by a factor of n/nk compared to random scatter.

• Are sensitive parameters helpful to find additional breaks?Monthly maximum temperature:

Due to the reduced spatial correlation Tmax “finds” less breaks.

Those identified are even better visible in Tmean.Drizzle parameter:

Highly inhomogeneous Many breaks found.But they do not coincide with breaks in mean precipitation.

• Vice versa we expect that Tmean breaks are helpful to find breaks in Tmax. But the prove of significance will be difficult.

detection of inhomogeneities in daily climate records to study trends in extreme weather detection...

Documents

bdaily stew meeting

solutiondaily stew meeting

high varext daily stew

daily climate records

real breaks

number of significant

maximum external variance

highest external variance