Slide 1

Eawag: Swiss Federal Institute of Aquatic Science and Technology Analyzing possible causes of bias of hydrological models with stochastic, time-dependent parameters Peter Reichert Eawag Dbendorf, ETH Zrich, and SAMSI Slide 2 SAMSI Transi- tion Workshop May 14-16, 2007 Contents Motivation Approach Implementation Application Discussion Motivation Approach Implementation Application Discussion Slide 3 SAMSI Transi- tion Workshop May 14-16, 2007 Motivation Approach Implementation Application Discussion Slide 4 SAMSI Transi- tion Workshop May 14-16, 2007 Motivation Typical results of a hydrological model Motivation Approach Implementation Application Discussion Overall quality of fit demonstrates that the model describes the most relevant mechanisms in the system adequately. However, remaining systematic deviations of model results from data make uncertainty analysis difficult. Slide 5 SAMSI Transi- tion Workshop May 14-16, 2007 Motivation Approach Implementation Application Discussion Problems Heteroscedasticity of residuals (even after Box-Cox transformation). Autocorrelation of residuals. Residuals of Box-Cox transformed results Slide 6 SAMSI Transi- tion Workshop May 14-16, 2007 Motivation Approach Implementation Application Discussion These problems are typical for any kind of deterministic dynamic environmental modelling. They make uncertainty analysis difficult as this can only be done if the statistical model assumptions are not seriously violated. Slide 7 SAMSI Transi- tion Workshop May 14-16, 2007 Motivation Suggested solution (Kennedy and OHagan, etc.) : Extend the model by a discrepancy or bias term. Replace: by: where y D = deterministic model, x = model inputs, = model parameters, y = observation error, B = bias or model discrepancy, Y M = random variable representing model results. Motivation Approach Implementation Application Discussion The bias term is usually formulated as a non-parametric statistical description of the model deficits (often as a Gaussian Stoachastic Process). Slide 8 SAMSI Transi- tion Workshop May 14-16, 2007 Motivation Advantage of this approach: The statistical description of the model discrepancy allows for improved uncertainty analysis. Disadvantage: Lack of understanding of the cause of the discrepancy makes it difficult to extrapolate. Motivation Approach Implementation Application Discussion We are interested in a technique that supports identification of the causes of model discrepancies. This can lead to an improved model formulation that reduces the discrepancies. This cannot be done by a purely statistical approach, but statistics can be supportive. Slide 9 SAMSI Transi- tion Workshop May 14-16, 2007 Motivation Causes of deficits of deterministic models: Errors in parameter values. Errors in model structure. Errors in model input. Inadequateness of a deterministic description of systems that contain intrinsic non-deterministic behaviour due to influence factors not considered in the model, model simplifications (e.g. aggregation, adaptation, etc.), chaotic behaviour. Motivation Approach Implementation Application Discussion Slide 10 SAMSI Transi- tion Workshop May 14-16, 2007 Motivation Pathway for improving models: 1.Reduce errors in deterministic model structure to improve average behaviour. 2.Add adequate stochasticity to the model structure to account for random influences. Motivation Approach Implementation Application Discussion This requires the combination of statistical analyses with scientific judgment. This talk is about support of this process by statistical techniques. Because of these deficits we cannot expect a deterministic model to describe nature appropriately. Slide 11 SAMSI Transi- tion Workshop May 14-16, 2007 Approach Motivation Approach Implementation Application Discussion Slide 12 SAMSI Transi- tion Workshop May 14-16, 2007 Approach Questions: 1.How to make a deterministic, continuous-time model stochastic? 2.How to distinguish between deterministic and stochastic model deficits? Motivation Approach Implementation Application Discussion Replacement of differential equations (representing conservation laws) by stochastic differential equations can violate conservation laws and does not address the cause of stochasticity directly. It seems to be conceptually more satisfying to replace model parameters (such as rate coefficients, etc.) by stochastic processes, as stochastic external influence factors usually affect rates and fluxes rather than states directly. Slide 13 SAMSI Transi- tion Workshop May 14-16, 2007 Approach Motivation Approach Implementation Application Discussion Slide 14 SAMSI Transi- tion Workshop May 14-16, 2007 Approach Note that the basic idea of this approach is very old. The original formulation was, however, limited to discrete-time systems with slowly varying driving forces (e.g. Beck 1987). Motivation Approach Implementation Application Discussion Our suggestion is to extend this original approach to continuous-time systems; allow for rapidly varying external forces; embed the procedure into statistical bias- modelling techniques. This requires more complicated numerical techniques and more extensive analyses of the results. Slide 15 SAMSI Transi- tion Workshop May 14-16, 2007 Implementation Motivation Approach Implementation Application Discussion Slide 16 SAMSI Transi- tion Workshop May 14-16, 2007 Model Deterministc model: Consideration of observation error: Motivation Approach Implementation Application Discussion Slide 17 SAMSI Transi- tion Workshop May 14-16, 2007 Model Model with parameter i time-dependent: Motivation Approach Implementation Application Discussion Slide 18 SAMSI Transi- tion Workshop May 14-16, 2007 Time Dependent Parameter This has the advantage that we can use the analytical solution: The time dependent parameter is modelled by a mean-reverting Ornstein Uhlenbeck process: or, after reparameterization: Motivation Approach Implementation Application Discussion Slide 19 SAMSI Transi- tion Workshop May 14-16, 2007 Inference We combine the estimation of constant model parameters,, with state estimation of the time-dependent parameter(s),, and with the estimation of (constant) parameters of the Ornstein-Uhlenbeck process(es) of the time dependent parameter(s),. Motivation Approach Implementation Application Discussion Slide 20 SAMSI Transi- tion Workshop May 14-16, 2007 Inference Gibbs sampling for the three different types of parameters. Conditional distributions: Ornstein-Uhlenbeck process (cheap) simulation model (expensive) Ornstein-Uhlenbeck process (cheap) Motivation Approach Implementation Application Discussion Tomassini et al. 2007 Slide 21 SAMSI Transi- tion Workshop May 14-16, 2007 Inference Metropolis-Hastings sampling for each type of parameter: Multivariate normal jump distributions for the parameters M and P. This requires one simulation to be performed per suggested new value of M. The discretized Ornstein-Uhlenbeck parameter,, is split into subintervals for which OU-process realizations conditional on initial and end points are sampled. This requires the number of subintervals simulations per complete new time series of. Motivation Approach Implementation Application Discussion Tomassini et al. 2007 Slide 22 SAMSI Transi- tion Workshop May 14-16, 2007 Estimation of Hyperparameters by Cross - Validation Due to identifiability problems we select the two hyperparameters ( ) by cross-validation: Motivation Approach Implementation Application Discussion Tomassini et al. 2007 Slide 23 SAMSI Transi- tion Workshop May 14-16, 2007 Estimation of Hyperparameters by Cross - Validation For a state-space model of the form we can estimate the pseudo-likelihood from the sample: Motivation Approach Implementation Application Discussion Tomassini et al. 2007 Slide 24 SAMSI Transi- tion Workshop May 14-16, 2007 Application Motivation Approach Implementation Application Discussion Application Slide 25 SAMSI Transi- tion Workshop May 14-16, 2007 Hydrological Model Simple Hydrological Watershed Model (1): Kuczera et al. 2006 Motivation Approach Implementation Application Discussion Slide 26 SAMSI Transi- tion Workshop May 14-16, 2007 Hydrological Model Simple Hydrological Watershed Model (2): Kuczera et al. 2006 1 2 3 4 5 7 8 A B 8 model parameters 3 initial conditions 1 standard dev. of obs. err. 3 modification parameters C Motivation Approach Implementation Application Discussion 6 Slide 27 SAMSI Transi- tion Workshop May 14-16, 2007 Hydrological Model Simple Hydrological Watershed Model (3): Motivation Approach Implementation Application Discussion Slide 28 SAMSI Transi- tion Workshop May 14-16, 2007 Model Application Data set of Abercrombie watershed, New South Wales, Australia (2770 km 2 ), kindly provided by George Kuczera (Kuczera et al. 2006). Box-Cox transformation applied to model and data to decrease heteroscedasticity of residuals. Step function input to account for input data in the form of daily sums of precipitation and potential evapotranspiration. Daily averaged output to account for output data in the form of daily averaged discharge. Motivation Approach Implementation Application Discussion Slide 29 SAMSI Transi- tion Workshop May 14-16, 2007 Analyses and Prior Distributions A) Estimation of constant parameters: Independent lognormal distributions for all parameters (8+3+1=11) with the exception of the measurement standard deviation (1/ ), keeping correction factors (f rain, f pet, f Q ) equal to unity. B) Estimation of time-dependent parameters: Ornstein-Uhlenbeck process applied to the log of the parameter. Hyperparameters: =1d, =0.2 (22%) fixed, only estimation of initial value and mean (0 for log f rain, f pet, f Q ). Constant parameters as above. Motivation Approach Implementation Application Discussion Slide 30 SAMSI Transi- tion Workshop May 14-16, 2007 Estimation of Constant Parameters A) Estimation of Constant Parameters: Try to find a reasonably good fit in which the deterministic model with constant parameters reproduces the major features of the data. The goal of the second analysis with time- dependent parameters will then be to support finding causes of remaining model deficiencies. Motivation Approach Implementation Application Discussion Slide 31 SAMSI Transi- tion Workshop May 14-16, 2007 Estimation of Constant Parameters Prior and Posterior Marginals: Motivation Approach Implementation Application Discussion Slide 32 SAMSI Transi- tion Workshop May 14-16, 2007 Estimation of Constant Parameters Max. post. simulation with constant parameters: Motivation Approach Implementation Application Discussion Slide 33 SAMSI Transi- tion Workshop May 14-16, 2007 Estimation of Constant Parameters Results of Constant Parameter Fit: The hydrological model with constant parameters leads to a fit that reasonably well reproduces the features shown by the data; a simulation with physically meaningful be- haviour of state variables with respect to their values and to their time scales of variation; identifiable model parameters (with the exception of the initial condition of h r ). Despite this basic agreement, the remaining systematic deviations violate simple statistical assumptions and make uncertainty analysis difficult. Motivation Approach Implementation Application Discussion Slide 34 SAMSI Transi- tion Workshop May 14-16, 2007 Estimation of Time-Dependent Parameters B) Estimation of time-dependent parameters: Sequentially replace constant parameters by time- dependent parameters. Try to learn from the results about deficits of the deterministic model structure as well as about the need for stochastic model extensions. Motivation Approach Implementation Application Discussion How to learn from the results of the analysis? 1.Analysis of temporal behaviour of parameters. 2.Analysis of posterior distributions of const. parameters. 3.Analysis of behaviour of model results. 4.Analysis of indicators of the quality of the fit. 5.Explorative analysis of the relationships between time- dependent parameters and system variables. Slide 35 SAMSI Transi- tion Workshop May 14-16, 2007 1. Temporal Behaviour of Parameters Time dependent parameter k_s Motivation Approach Implementation Application Discussion Slide 36 SAMSI Transi- tion Workshop May 14-16, 2007 1. Temporal Behaviour of Parameters Time dependent parameter f_rain Motivation Approach Implementation Application Discussion Slide 37 SAMSI Transi- tion Workshop May 14-16, 2007 1. Temporal Behaviour of Parameters Time dependent parameter f_Q Motivation Approach Implementation Application Discussion Slide 38 SAMSI Transi- tion Workshop May 14-16, 2007 1. Temporal Behaviour of Parameters Time dependent parameter s_F Motivation Approach Implementation Application Discussion Slide 39 SAMSI Transi- tion Workshop May 14-16, 2007 1. Temporal Behaviour of Parameters Assessment: In cases with highly dynamic external forcing, identified parameter time series are difficult to interpret directly. The variation of width measures of the posterior time-dependent parameter allows us to distinguish time periods during which we can gain information about variations in the parameter from periods during which we cannot. In our example, this varies somewhat from one parameter to the other, with a general tendency that we can learn more during periods with rain events than during dry weather periods. Motivation Approach Implementation Application Discussion Slide 40 SAMSI Transi- tion Workshop May 14-16, 2007 2. Posterior of Constant Parameters Results for time dependent parameter k_s Motivation Approach Implementation Application Discussion Slide 41 SAMSI Transi- tion Workshop May 14-16, 2007 2. Posterior of Constant Parameters Results for time dependent parameter f_rain Motivation Approach Implementation Application Discussion Slide 42 SAMSI Transi- tion Workshop May 14-16, 2007 2. Posterior of Constant Parameters Results for time dependent parameter f_Q Motivation Approach Implementation Application Discussion Slide 43 SAMSI Transi- tion Workshop May 14-16, 2007 2. Posterior of Constant Parameters Results for time dependent parameter s_F Motivation Approach Implementation Application Discussion Slide 44 SAMSI Transi- tion Workshop May 14-16, 2007 2. Posterior of Constant Parameters Assessment: The marginal posterior distributions of some parameters depend significantly on which of the parameters was made time-dependent. In particular, making the modification factor for rain intensity time dependent, changes the posterior distributions of the other parameters strongly. This demonstrates the importance of addressing input (rainfall) intensity carefully. Motivation Approach Implementation Application Discussion Slide 45 SAMSI Transi- tion Workshop May 14-16, 2007 3. Behaviour of Model Results Results for time dependent parameter k_s Motivation Approach Implementation Application Discussion Slide 46 SAMSI Transi- tion Workshop May 14-16, 2007 3. Behaviour of Model Results Results for time dependent parameter f_rain Motivation Approach Implementation Application Discussion Slide 47 SAMSI Transi- tion Workshop May 14-16, 2007 3. Behaviour of Model Results Results for time dependent parameter f_Q Motivation Approach Implementation Application Discussion Slide 48 SAMSI Transi- tion Workshop May 14-16, 2007 3. Behaviour of Model Results Results for time dependent parameter s_F Motivation Approach Implementation Application Discussion Slide 49 SAMSI Transi- tion Workshop May 14-16, 2007 3. Behaviour of Model Results Assessment: The basic features of the solutions are not changed by introducing a time-dependent parameter. For some of the parameters, making them time- dependent significantly reduces the bias in model output, for others this is not the case. Motivation Approach Implementation Application Discussion Slide 50 SAMSI Transi- tion Workshop May 14-16, 2007 4. Quality of Fit Improvement with time-dependent parameters: Motivation Approach Implementation Application Discussion Nash-Sutcliffe indices: k s 0.83 f rain 0.78 f Q 0.68 s F 0.64 k r 0.59 f pet 0.57 q gw,max 0.54 q lat,max 0.53 k dp 0.53 k bf 0.53 base0.53 Assessment: Input (f rain ) and out- put (f Q ) corrections. Potential for soil / runoff model (k s, S F ) improvements. Some potential for river and evaporation improvements. Random or deterministic? Slide 51 SAMSI Transi- tion Workshop May 14-16, 2007 5. Relationsship with Model Variables Scatter plot of k_s vs. model variables Motivation Approach Implementation Application Discussion Slide 52 SAMSI Transi- tion Workshop May 14-16, 2007 5. Relationsship with Model Variables Scatter plot of f_rain vs. model variables Motivation Approach Implementation Application Discussion Slide 53 SAMSI Transi- tion Workshop May 14-16, 2007 5. Relationsship with Model Variables Scatter plot of f_Q vs. model variables Motivation Approach Implementation Application Discussion Slide 54 SAMSI Transi- tion Workshop May 14-16, 2007 5. Relationsship with Model Variables Scatter plot of s_F vs. model variables Motivation Approach Implementation Application Discussion Slide 55 SAMSI Transi- tion Workshop May 14-16, 2007 5. Relationsship with Model Variables Motivation Approach Implementation Application Discussion Assessment: Most of the time dependent parameters do not show deterministic variation with any of the system variables. The only exception is the parameter k s of the soil submodel that varies significantly with the saturated area (which it parameterizes). Slide 56 SAMSI Transi- tion Workshop May 14-16, 2007 Conclusions Motivation Approach Implementation Application Discussion Assessment: Stochasticity seems to be the dominating cause of deviations of model results from measurements. This is likeli to be dominated by input (rainfall) uncertainty. The highest chance to find an improvement of the deterministic model is for the soil/runoff submodel of the hydrological model. It seems difficult to significantly improve the model by changes to the groundwater and river sub-models. Slide 57 SAMSI Transi- tion Workshop May 14-16, 2007 Hydrological Model Model extensions: Motivation Approach Implementation Application Discussion Extension 1: Modification of runoff flux: Extension 2: Modification of sat. area funct.: Both extensions lead to three more model parameters. Slide 58 SAMSI Transi- tion Workshop May 14-16, 2007 Hydrological Model Previous results: Motivation Approach Implementation Application Discussion Nash-Sutcliffe indices: k s 0.83 f rain 0.78 f Q 0.68 s F 0.64 k r 0.59 f pet 0.57 q gw,max 0.54 q lat,max 0.53 k dp 0.53 k bf 0.53 base0.53 Extended models: Nash-Sutcliffe indices: ext. 10.72 ext. 20.54 Assessment: Model extension 1 significantly improves the description of the system. Slide 59 SAMSI Transi- tion Workshop May 14-16, 2007 Approach Motivation Approach Implementation Application Discussion Slide 60 SAMSI Transi- tion Workshop May 14-16, 2007 Hydrological Model Next Steps Redo analysis with model extension 1. Compare remaining stochastic uncertainty with knowledge on input uncertainty. Do uncertainty analysis for model with extensions 1 and input uncertainty. Investigate alternative ways of describing rainfall input uncertainty Conclusions The application of the technique led to the dis- covery of improvements of the deterministic model structure as well as to the inclusion of stochasticity. Motivation Approach Implementation Application Discussion Slide 61 SAMSI Transi- tion Workshop May 14-16, 2007 Discussion Motivation Approach Implementation Application Discussion Slide 62 SAMSI Transi- tion Workshop May 14-16, 2007 Discussion The suggested procedure seems to fulfill the expectations of supporting the identification of model deficits and of introducing stochasticity into a deterministic model. There is need for future research in the following areas: Explore alternative ways of learning from the identified parameter time series. Different formulation of time-dependent parameter (for some applications smoother behaviour). Improve efficiency (linearization, emulation). Learn from more applications. Motivation Approach Implementation Application Discussion Slide 63 SAMSI Transi- tion Workshop May 14-16, 2007 Transition On-going projects in various fields: Applications for gaining more experience: Reichert et al.:hydrological model Cintron et al.:epidemiological model Emulation of dynamic models: Reichert et al.:simple physical-based prior White et al.:extended physical-based prior Liu et al.:statistical prior Gosling et al.:emulation of time step Linearization for improving efficiency: Paulo et al.:emulation of linearized model Liu et al.:direct use of linearized model Other persons: Bayarri, Santner, Pitman, OHagan, Wolpert More ideas on the way. Post-program workshop next year? Motivation Approach Implementation Application Discussion Slide 64 SAMSI Transi- tion Workshop May 14-16, 2007 Acknowledgements Development of the technique: Hans-Rudolf Knsch, Roland Brun, Lorenzo Tomassini, Mark Borsuk, Christoph Buser. Hydrological example Johanna Mieleitner, George Kuczera. Interactions at SAMSI: Susie Bayarri, Tom Santner, Gentry White, Ariel Cintron, Fei Liu, Rui Paulo, Robert Wolpert, John Paul Gosling, Tony OHagan, Bruce Pitman, Jim Berger, and many more. Motivation Approach Implementation Application Discussion I would like to thank in particular to Jim Berger and Susie Bayarri for setting up this program that lead to a very stimulating and fruitful stay for me at SAMSI.