calibration of computer simulators using emulators

Calibration of Computer Simulators using Emulators

Recap –Emulators

• We are concerned with complex, non-linear simulators

• In this session we will look at calibration of such simulators

• We will heavily depend on emulators • An emulator is a Gaussian process (or second order

process) that interpolates the simulator output• Emulators are fast

EGU short course - session 4 2

Calibration

• Simulator users often want to tune the simulator using observations of the real system

• Adjust the input parameters so that the simulator output matches observations as well as possible

• Two very important points1. Calibration will reduce uncertainty about x but will not

eliminate it2. It is necessary to understand how the simulator relates to

reality• Model discrepancy


Calibration and Assimilation

• Calibration is concerned with the values of the inputs that are consistent with the data.

• Assimilation is concerned with producing the best forecast/hindcast

• Calibration changes the simulator inputs• Assimilation changes the simulator state variables


Model discrepancy

• Simulator output y = f(x) will not equal the real system value z

• Even with best/correct inputs x• Model discrepancy is the difference z – f(x)• As discussed in Session 1, model discrepancy is due to

• Wrong or incomplete science• Programming errors, rounding errors• Inaccuracy in numerically solving systems of equations

• Ignoring model discrepancy leads to poor calibration• Over-fitting of parameter estimates• Over-confidence in the fitted values


History matching

• History matching (a term taken from the petroleum industry) means finding sets of inputs that given simulator outputs that are ‘compatible’ with data

• Calibration means finding a best value (or a distribution) for the inputs given the data


Implausibility

• Define a measure of implausibility (Imp)

• If the implausibility is greater then ±3 those values of the inputs are deemed implausible

• Because this is a function of the emulator not the original simulator runs we calculate it everywhere in input space


Waves of Implausibility

• Wave 1: Apply the implausibility measure. Mark part of input space as implausible

• Wave 2: Add extra points in the not implausible region and rebuild the emulator. Repeat the implausibility measure

• Wave 3+: Repeat until the implausible region ceases to grow

EGU short course - session 4

8

A 1-d example


Example -Galform


Example - Galform• Galform is a simulator of Galaxy formation• It has 17 inputs• The amount of not implausible space in each wave is

• None of the original 1000 member LHC was an acceptable fit to the data


Wave 1 14.9%

Wave 2 5.9%

Wave 3 1.6%

Wave 4 0.26%

Wave 5 0.036%

Calibration

• In history matching we were simply looking for regions of input space that were not implausible given the data.

• In calibration we want to find the ‘best input’ x (and it associated uncertainty)


Kennedy and O’Hagan(2001)• ζ is the real system• z=ζ+ε is data on the real system (ε~N(0,σ2))• y=f(x) is the simulator output• d=ζ-y is the model discrepancy

• ζ=f(x)+d• Build an emulator for f and simultaneously model

the discrepancy as a GP• ζ=f*(x)+d*

• z=f*(x)+d*+εEGU short course - session 4 19

Kennedy and O’Hagan (2001) -2

• We can now perform an uncertainty analysis• This shows how much we have learned about the

simulator inputs from the data• The mean/mode of the posteriors give us our

estimate of the best value for the inputs


Model Discrepancy Revisited

• We have seen that we can use the model discrepancy to calibrate/history match the simulator

• We can also look at the discrepancy between different simulators

• This is particularly interesting if we have hierarchies of simulators


Hierarchies of Simulators

• Often we have hierarchies of simulators• Usually the resolution is increasing but additional

processes could be added


Hierarchies of Simulators

• Rather than emulate each simulator separately• Emulate simulator 1 and then emulate the difference

between each level• Need to have some runs at common inputs• Need few runs of expensive complex simulators


Reified Simulators


Reified Simulators

• Reified simulators are ‘imaginary’ simulators that we impose between our simulators and reality

• They are the ‘best’ simulator we could produce• Model discrepancy is split into two:

1. The discrepancy between the current simulator and the reified simulator

2. The discrepancy between the reified simulator and reality

• Reification does not reduce the discrepancy. It might make it easier to elicit.


Overview

• Emulators are useful tools in the calibration of complex simulators

• Two methods have been described:• History Matching – ruling out implausible regions of input

space• Calibration – Finding ‘best fit’ input values

• Reification may be useful in eliciting the relationship between simulators and reality


calibration of computer simulators using emulators

Documents

growegu short course

dataegu short course

fastegu short course

simulator inputsassimilation

simulator outputemulators

simulator outputs

original simulator

inputs xmodel discrepancy