long term weather and flux data: treatment of discontinuous data
DESCRIPTION
Long term weather and flux data: treatment of discontinuous data. Bart Kruijt, Wilma Jans, Cor Jacobs, Eddy Moors. Loobos. Gap filling – meteorologidal data. - PowerPoint PPT PresentationTRANSCRIPT
Long term weather and flux data: treatment of discontinuous data.
Bart Kruijt, Wilma Jans, Cor Jacobs, Eddy Moors
Loobos
Gap filling – meteorologidal data
• Gap filling is a grey area between measurement, statistics and modelling. We should be careful not to ‘double model’: use filled data for calibration, validation, etc. Should we not go for just modelling?
• There is a need for continuous data• fluxes:
– Integration over time of fluxes, with estimate of uncertainty, needs gaps filled with correct mean and sd distribution needs to be correct
• Meteo: – Models need updating of state variables (soil moisture, biomass)– Total radiation, rainfall, means of T, Rh, U etc need to be correct
• EU – GEOLAND project required gap-filled meteo data for 2003, to test-run 1-D surface-atmosphere models.
• Particular to meteo data:– Meteo vars often are poorly correlated with other variables– Often, if one variable is missing, most others are as well
• Therefore, either use internal variability, autocorrelations, or
• Use correlations with data measured nearby
Are conditions for grass and forest stations the same?
temperature (oC)
-10 0 10 20 30
fore
st 2
-10
0
10
20
30temperature (oC)
10 15 20 25 30 35
gras
s
10
20
30
40
rel. humidity (%)
0 20 40 60 80 100
gras
s
0
20
40
60
80
100
wind speed (m s-1)
forest
0 5 10 15gr
ass
0
5
10
15
rel. humidity (%)
0 20 40 60 80 100
fore
st 2
0
20
40
60
80
100
wind speed (m s-1)
forest 1
0 5 10 15
fore
st 2
0
5
10
15
Foresty0 a r2
Temperature Grass -1.01 1.016 0.858Forest 2 0.489 0.960 0.990
Rel. humidity Grass -5.847 1.007 0.835Forest 2 18.99 0.811 0.897
Wind speed Grass 0.308 1.365 0.359Forest 2 0.036 1.015 0.747
Neural network (multiple non-linear regressor):
xey
1
21 hidden layer
x1 x2 x3 x4 input signals
output signaly
Activation functionhidden layer:
Input scaled between -1 and 1
Neural network configuration to estimate Lin:
NN calibrated on: Lin - T4
Input variables Hiddenneurons
CalibrationRMSE
ValidationRMSE
TofD, Sin Top Atm 3 30.70 31.42TofD, Sin Top Atm, Lin clear sky, Rh, Tair, u 4 24.19 25.49cldcover 2 25.78 26.20cldcover, Rh, Tair 2 23.68 24.37cldcover, Rh, Tair 6 21.45 24.95TofD, Sin Top Atm, H, u*, (z-d)/L 4 30.30 30.53Sin Top Atm, H, u*, (z-d)/L 3 25.08 26.86Sin, cldcover, Rh, Tair, H 3 20.48 21.18Sin, Lin clear sky, cldcover, Rh, H, u* 4 18.30 19.74Sin Top Atm, Sin, Lin clear sky, cldcover, Rh, Tair, u, H, u* 4 17.32 19.50
Long wave incoming radiation (Validation):
• Lin clear sky:slope = 1.122 r2 = 0.27
• Lin neural netslope = 0.985 r2 = 0.67
Lin (W m-2)
measured
200 250 300 350 400 450 500
mod
elle
d
200
250
300
350
400
450
500
clear skyNNregression
Uncertainty and the length of the data gap:
Long wave incoming radiation
missing data (%)
0 20 40 60 80 100
RM
SE
16
18
20
22
24
26
28
30
32
34
missing datacalibration data
Neural network configuration to estimate F_CO2:
Input variables RMSEλE 67.25Sin, Tair, SM, Tsoil 39.80Sin, Rh, Tair, SM, Tsoil, 40.35λE, Sin, Tair, SM, Tsoil 38.20λE, Sin, Tair, SM, Tsoil, Interception 38.54λE, Sin, Tair, SM, Tsoil, pair 38.77λE, Lin, Rh, Tair, u, SM, Tsoil, pair 44.81λE, Sin Top Atm, Sin, Lin, Rh, Tair, u, SM, Tsoil, pair 37.48
• Fill missing data AWS• Fill missing data latent heat flux• Fill missing data CO2 flux
• Neural networks are useful as they can combine correlations with any internal or external data, and make few assumptoins
• However, setting up NN for individual sites can be time consuming (Moors method) and using external data also (convert, standardise, link )
‘perverted’ CE method (CE= web-based tool Reichstein&Papale)• We are usually in a hurry and needed only ‘reasonable’ results
• We discovered: CE method accepts any data series as input in any of the filling columns! – NEE (and other fux) columns are correlated with T, Rad columns– T, Rad columns are also filled
• We thought we might use this as an easy, lazy way to fill gaps in meteo data!– Assumes the methis is a purely statistical tool
• We applied the method to create continuous data for GEOLAND, for several FLUXNET sites– For T, Rad, Rh, P, Precip!– the result looks acceptable.
• We tested this putting in T, Rad or U data in NEE column– Created artifical gasp in loobos data– Compared with NN gap filling and original data
Hungary – Hegygatsal – Temperature filled
Hegyhatsal – Specific humidity !
Tharandt windspeed Soroe rainfall
Results Loobos test: data, neural network, CE filling: LE
day 48-55
-100
0
100
200
300
Day
LE
(W
.m-2)
CE
NN
Measured
day 100-107
-50
0
50
100
150
200
250
300
350
Day
LE
(W
.m-2)
day 170-175
-100
0
100
200
300
400
500
Day
LE
(W
.m-2)
day 220-225
-100
0
100
200
300
400
500
Day
LE
(W
.m-2)
day 260-265
-100
0
100
200
300
400
500
Day
LE
(W
.m-2)
day 350-355
-100
0
100
200
300
Day
LE
(W
.m-2)
Results: data, neural network, CE filling: NEEday 48-55
-30
-20
-10
0
10
20
Day
NE
E (
µMo
l.m-2.s
-1)
CE
NN
Measured
day 100-107
-30
-20
-10
0
10
20
Day
NE
E (
µMo
l.m-2.s
-1)
day 170-175
-30
-20
-10
0
10
20
Day
NE
E (
µMo
l.m-2.s
-1)
day 220-225
-30
-20
-10
0
10
20
Day
NE
E (
µMo
l.m-2.s
-1)
day 260-265
-30
-20
-10
0
10
20
Day
NE
E (
µMo
l.m-2.s
-1)
day 350-355
-20
-10
0
10
20
Day
NE
E (
µMo
l.m-2.s
-1)
Compare filled totals (Monthly NEE)
monthly totals
-140-120-100-80-60-40-20
02040
-150 -100 -50 0 50
measured data (gC m-2mo-1)
arifi
cial
gap
s fil
led
(gC
m-
2mo-
1)
carboEurope fill
Neural Network
Results: data, neural network, perverse CE filling
day 48-55
-5
0
5
10
15
Day
T
CE
NN
Measured
day 100-107
0
5
10
15
20
25
Day
T
day 170-175
0
5
10
15
20
25
Day
T
day 220-225
0
5
10
15
20
25
30
35
Day
T
day 260-265
0
5
10
15
20
25
Day
T
-Temperature-Five 6-8 day gaps
Results: data, neural network, perverse CE fillingday 48-55
0
100
200
300
400
500
600
Day
Sin
(W
/m2)
CE
NN
Measured
day 100-107
0
200
400
600
800
1000
Day
Sin
(W
/m2)
)
day 170-175
0
200
400
600
800
1000
Day
Sin
(W
/m2)
day 220-225
0
200
400
600
800
1000
Day
Sin
(W
/m2)
day 260-265
0
200
400
600
800
Day
Sin
(W
/m2)
-Shortwave radiation-Five 6-8 day gaps
Results: data, neural network, perverse CE filling
day 48-55
0
20
40
60
80
100
120
Day
Rh
(%
)
CE
NN
Measured
day 100-107
0
20
40
60
80
100
Day
Rh
(%
)
day 170-175
0
20
40
60
80
100
120
Day
Rh
(%
)
day 220-225
0
20
40
60
80
100
120
Day
Rh
(%
)
day 260-265
0
20
40
60
80
100
Day
Rh
(%
)
-Relative humidity-Five 6-8 day gaps
Results: data, neural network, perverse CE filling
-Wind speed-Five 6-8 day gaps
day 48-55
0
1
2
3
4
5
6
7
Day
U (
m/s
)
CE
NN
Measured
day 100-107
0
1
2
3
4
5
Day
U (
m/s
)
day 170-175
0
1
2
3
4
5
6
7
8
Day
U (
m/s
)
day 220-225
0
1
2
3
4
5
6
Day
U (
m/s
)
day 260-265
0
1
2
3
4
5
6
7
8
Day
U (
m/s
)
Conclusions:
• Also work on filling Meteorology data– For Meteo data the Perverse CE does not perform very well
after all (in representing variability and pattern.– Filling in winter is more difficult than in summer – NN is good at representing pattern and variability, but mean
can be biased
• Future: develop NN methods, including– Correlate with ECMWF reanaysis data. Partly with the
reanalysis product, partly with the forecast product (rainfall). 3- to 6 hourly data.
– Possibly use measured data for rainfall– Produce filled series for many towers centrally.
• …………
Jaru 50%-100%
Ave
rag
e d
aily
ca
rbo
n flu
x (T
ha
-1 d
-1)
-0.04
-0.02
0.00
0.0210-day average daily total fluxfit to only even 10-day periodsfit to only odd 10-periods
Col 61 vs fit tots
Jaru 25%
Jan 99 Jul 99 Jan 00 Jul 00 Jan 01
Ave
rage
da
ily c
arb
on
flu
x (T
ha
-1 d
-1)
-0.04
-0.02
0.00
0.02 10-day average daily total fluxfit to every 2nd in 4 10-day periodsfit to every 4th in 4 10-day periodsfit to every 1st in 4 10-day periodsfit to every 3rd in 4 10-day periods
Jaru 12.5%
Jan 99 Jul 99 Jan 00 Jul 00 Jan 01
10-day average daily total fluxfit to every 1st in 8 10-day periodsfit to every 2nd in 8 10-day periodsfit to every 3rd in 8 10-day periodsfit to every 4th in 8 10-day periods
Uncertainty as a function of the percentage good data - Rebio Jaru
Percentage annual data coverage
0 10 20 30 40 50 60 70 80 90 100
Cum
ulat
ive
stan
dard
err
or o
f est
imat
e
(T h
a-1y-1
)
0.0
0.5
1.0
1.5
2.0
2.5
Number of full data days per year
0 50 100 150 200 250 300 350
JaruManaus K34
Manaus K34
Jun 99 Dec 99 Jun 00 Dec 00
Ave
rag
e d
aily
C f
lux
(t h
a-1 d
-1)
-0.04
-0.03
-0.02
-0.01
0.00
0.01
0.02
Fit and 95% confidence of fitdata eddy flux (10 day averages)
Rebio Jaru 1999-2000
Jan 1999 Jul 1999 Jan 2000 Jul 2000 Jan 2001
Ave
rag
e d
aily
carb
on f
lux
(t h
a-1 d
-1)
-0.04
-0.02
0.00
0.02
Eddy flux data (10 day averages)Fit to all data
Seasonal and interannual variation of net daily carbon fluxes
Less seasonal
More seasonal
Total uptake % data gaps 95% confidence intervalManaus Jan'00 - Jan '01 6.2 T ha-1 5%-10% - not specifiedManaus Jul'99 - Jul'00 7.7 T ha-1 17 % +/- 0.25 T ha-1
Jaru March '99-March '00 5.8 T ha-1 57% +/- 1.0 T ha-1
Jaru Oct. '99 - Oct '00 6.0 T ha-1 40% +/- 0.7 T ha-1
U* • lm Fc=f(C,u*,lm,R,Ps)
Advection=f(C)Advection
Consider the area beneath the sensor a leaky, sloshing vesseland fit both physiological and micrometeorological parameters
R, Ps=alpha.PAR
To be tested ….
C=sum(R-Ps-Fc-advection)