www.kcl.ac.uk

End user understandings of uncertainty, ensemble

forecasts, and EFAS

Sebastien Nobert, David Demeritt, Hannah Cloke Department of Geography

www.kcl.ac.uk

Outline of Talk Today

1. Background to Study 2. General understandings of ensembles

and uncertainty in flood forecasting 3. Responses to EFAS in particular 4. Challenges to using EPS 5. Conclusions & recommendations

www.kcl.ac.uk

1. Background: Preliminary findings from a larger study

‘Europeanizing flood forecasting and the geographies of risk and science in the EU’ (RES-062-23-0913)

Project Aims: 1. Explain the institutional politics of flood and disaster

management in Europe; 2. Document the beliefs and practices involved in flood

forecasting and explore their implications for managing floods in the face of uncertainty;

3. Enrich the understanding and communication of ensemble predictions (EPs) and advise EFAS about improving its specific role in flood risk assessment.

www.kcl.ac.uk

Promises of Ensemble Flood Forecasts

“the use of meteorological ensembles to produce sets of hydrological predictions increased the capability to issue flood warnings” (Balint et al., 2006, p.67)

“The hydrological ensemble predictions have greater skill than deterministic ones”. (Roulin, 2007)

“The use of EPS in hydrological forecasting proved to be of great added value to a flood early warning system, as the EPS-based forecasts showed in general higher skill than the deterministic-based ones”. (Bartholmes et al., 2008)

Are these technical promises realized in operational practice?

www.kcl.ac.uk

Research Design and Evidence Base

• 45+ interviews with forecasting experts, water authorities and CPAs in 14 European countries involved in flood forecasting, warning and emergency response

• Participant observation at forecasting workshops & conferences and during visits to forecasting centres

• All interview quotations are anonymized

“what we call our data are really our constructions of other people’s constructions of what they are up to” (Geertz 1973: 9)

www.kcl.ac.uk

Methodology: Qualitative Social Science

Not an experimental science in search of laws; an interpretative one in search of understanding, or Verstehen

Different objects of study, different methodologies – Indifferent kinds (ie.

rocks and other unconscious objects of study)

– Interactive kinds (people: objects who are also conscious subjects)

Natural science

Social science

www.kcl.ac.uk

Seeking a ‘Thick Description’ of Meaningful Behaviour and Belief

Consider two boys rapidly contracting the eyelids of their right eyes. In one this is an involuntary twitch; in the other a conspiratorial signal to a friend. The two are, as movements, identical... Yet the difference, however, unphotographable, between a twitch and wink is vast; as anyone unfortunate enough to have had the first taken for the second knows…[T]he point is that between … the ‘thin description’ of what the winker/twitcher is doing (‘rapidly contracting his right eyelid’) and the ‘thick description’ of what he is doing (‘practicing a burlesque of a friend faking a wink to deceive an innocent into thinking a conspiracy is in motion’) lies the object of ethnography: a stratified hierarchy of meaningful structures in terms of which twitches, winks, fake-winks, parodies, rehearsals of parodies are produced, perceived, and interpreted.”– Geertz (1973)

www.kcl.ac.uk

Accessing End User Understandings

• Solicited for broad interviews about operational flood warning and response – EFAS not specifically mentioned

• Promised anonymity • Enthusiastic to show me

around • My ‘positionality’:

– independent of JRC: allows more frank assessment of EFAS

– Scientific novice: requires informants to explain

www.kcl.ac.uk

Interview Protocol & Questions • Semi-structured

– Broad set of topics & prompts – Flexibility to adapt to flow of conversation – Interviewees use own words and can steer conversation

• Topics – Describe your flood forecasting, warning and emergency

response system – forecasting and data systems used – Warning protocols and communication – CPA responsibilities and responses – Attitudes to uncertainty and EPS – Knowledge & Experience EFAS

• Asking about EFAS at the start of the interview would have resulted in very different answers

www.kcl.ac.uk

Outline of Talk Today

1. Background to Study 2. General understandings of ensembles

and uncertainty in flood forecasting 3. Responses to EFAS in particular 4. Communicating EPS and uncertainty 5. Conclusions & recommendations

www.kcl.ac.uk

2. EPS: an often unfamiliar product Differing levels of awareness

“Well, the EPS is rather, um, is rather rare, from what I’ve heard. Most countries really work with deterministic”

“I've seen products from EFAS several

times at meetings, at workshops. I don’t get it regularly. So I can't comment on the operational use or something like that.... I would be interested in having this product and looking at that ... that they can give a more profound ... understanding”

Some surprising ignorance, even among MoU signatories: “ EPS, what does that mean? Issue of the model ensembles is it makes some average number of when you have few different models and this is the resulting model. Is it right? Some average? “

www.kcl.ac.uk

Informational Value of EPS Contested

• Ensemble mean= the ‘best guess’ forecast? “But generally, it's better to have these ensembles.

Even if you don’t use the probabilistic information, just use the ensemble mean. This is basically what we do for some customers. We give them ensemble mean because it's better than the deterministic one”.

• Ensemble mean=meaningless statistic “This is still an important debate. So there are

some people around the world trying to use the mean. If you use the mean, it's not too much value.”

www.kcl.ac.uk

Using EPS as a check on local deterministic model?

Forecasters sometimes looked to EFAS as a ‘check’ on their own deterministic models: “Forecaster: We are linked, in the EFAS family and so we compare our

data. Interviewer: With their predications? Forecaster: Yeah, we use EFAS.... we usually check it if we see the

situation is dangerous or might be dangerous by our own judgement ...” “Interviewer: Let's say you're using your model and then you look at what

EFAS is predicting . . . Forecaster: Yeah, we can compare and we can decide and we can strictly

express that it's true or not . . . [whether] our statements are right or wrong…”

EPS not designed as validation tool

www.kcl.ac.uk

Can EPS improve flood risk management? Forecasters complain CPAs can’t

tolerate uncertainty or error: “At the very end, I think everything must be a kind

of deterministic forecast ... Does lake exceed this threshold, yes or no? It's always . . . at the very end, it's always a yes or no decision.”

Others welcome probabilistic information: “it's presented in percentage ... That’s really

good because ...I can see,’Well, I have just this amount of man or equipment and so on, where shall I put it?’ If I see it like a likelihood, then I can take my own decision.”

www.kcl.ac.uk Time

1. uncertainty about rainfall? “What we would like to see is a better precision in rainfall forecasts. This is because the main source of uncertainty in forecasting floods comes from the rain scenarios”

Dispute over critical forecasting uncertainties to resolve

2. uncertainty about runoff processes? “In hydrology, we are obviously conscious of our model uncertainty. What is more uncertain than hydrological models? I don’t really know frankly, but it is obvious that hydrological models are as uncertain as meteorological models. This is because the representation of river basin is so simplified in comparison with reality”

3. Quantifying total uncertainty? “I said if a forecast is 5 spaghetti lines above our threshold, it doesn't mean that the probability is 10%...it might be 5% or 50%. I don’t know. I can't estimate it.”

www.kcl.ac.uk

Broad patterns in understandings of EPS and flood forecast uncertainty

• Still working to quantify open-ended interview data • No consistent differences in attitudes between

meteorologists and hydrologists • Clear contrasts between ‘old EU’ states and new

accession & EU candidate countries – Wealthy EU states often have or developing own EPS

capacity for NWP and/or flood forecasting – New accession states have fewer resources & other

development priorities: better instrumentation, more computing, training, etc

“Meteorological and hydrological forecasting cascaded or state of art prediction model for select profiles. This is our dream. This is our big . . . how to say? We want very much to do something like that, but have some obstacles. . . You need very good people, very good in mathematics, very good in modelling and very good in programming. But those people are very expensive. Salary in our institute is …not very attractive for those people.”

www.kcl.ac.uk

Outline of Talk Today

1. Background to Study 2. General understandings of ensembles

and uncertainty in flood forecasting 3. Responses to EFAS in particular 4. Communicating EPS and uncertainty 5. Conclusions & recomendations

www.kcl.ac.uk

3. Responses to EFAS in particular

Generally welcomed by forecasters as a useful pre-alert: ”Yeah. We also get EFAS reports. For us, it's a useful pre information,

but normally 2 or 3 or 4 days before a flood, we normally know from the weather forecast that we should be aware of this...”

Summary tables a good way to show ensemble “But I have the impression with the presentation of these boxes which shift

every new round is quite a good way of presenting uncertainty in the forecasts”

Widespread desire to see hydrographs as part of alerts

“If they would give a hydrograph, forecast with hydrograph, that would be…fantastic!”

www.kcl.ac.uk

EFAS complements national capacities

Medium term warnings operationally useful to forecasting centres “It is useful because these warnings came to our office

let's say 4 or 5 days before. ... it's useful for us to raise the preparedness of our staff a few days before.

A few saw EFAS pre-alerts as useful for allowing earlier warnings to CPAs “it’s definitely very useful, the direct operational

relevance is for the moment in terms of civil protection.

www.kcl.ac.uk

EFAS works best on big rivers

Forecasters and CPAs are not especially troubled by forecasting those risks

“The other thing is it's very, very dependent on the size of the catchment, the quality of the forecasts. If you do forecast for large rivers, it's more easy than if you have some torrents. If you have in the alpine region, you can't do forecasting there.” “CPA: Danube, it's very well-known. So

it's very slow from rising and falling. Interviewer: But for small rivers though it

must be more difficult? CPA: The problems there are really

dangerous. You can't see . . . may be you have small scale thunderstorms. It's not possible to give a prognosis in about 30 minutes.”

www.kcl.ac.uk

Flash flooding perceived as bigger threat, but EFAS less help

“The problem [here] is these rivers are alpine type, very small scale and [EFAS], it's good information.. . . but not for detailed flood forecasting... It's a rough idea for someone sitting in Brussels and in London. But not really useful for someone sitting [here]. This is just a problem of scale”

www.kcl.ac.uk

Concerns: what do EFAS warnings mean?

Hydrographs of river flow or water levels the traditional measure of flood risk in hydrology

EFAS alerts based on # ensemble members exceeding unspecified thresholds

“The problem with EFAS particularly is that …they do not show hydrographs, but they only tell you that there is a chance of serious flooding…But they don’t say what the level will be. They only say there is a chance of serious flooding or extreme flooding, but you don’t know what it means.”

Number of EFAS ensemble members exceeding the ‘high’ (HAL) and ‘serious’ (SAL) alert levels

www.kcl.ac.uk

Thresholds: too low, confusion about how they are calculated

“[...] the question is how EFAS thresholds are established. This is our big problem since the thresholds of our two models are tailored to our data bank. [...] They have simply taken the 20 years of flow simulation since 1995-98. Thus, because these years were relatively dry, their thresholds are very very low.

“Forecaster: So far, most alerts have been a little bit too early. Interviewer: Do you think it’s because of EPS system, or… Forecaster: No, because their warning levels are too low. Interviewer: Do you know what they use as thresholds? Forecaster: They use observations, they look at the data set that they

have in some cases and this is a very limited data set of about five six years. And they look at the maximum level that has occurred in these periods.”

www.kcl.ac.uk

Broad patterns of response: new accession states most consistently enthusiastic

“very important I think is this initiative of EFAS project. It's useful one. And once they improve their models, it will be [even better].... So anyway, we are a partner in this cooperation and we receive their product. It's another source of getting information of possibility of having flood events [here]”

– For scientific reasons... “This is quite comfortable for us because they [EFAS] have very good

information” Back door access to higher quality ECMWF ensemble products

– For internal organizational reasons... • Participating in EFAS also about being ‘European’ • Provides leverage for securing additional resources • Access to international scientific networks

www.kcl.ac.uk

‘Old’ Europe less enthusiastic

– EFAS one of many, many sources of information

“It is useful. But we could easily live without it.”

– Sometimes think local systems superior “They coordinate I think from Ispra from Italy. Yeah, I've heard

of it. But the colleagues ...said it's not so good than our model.”

– Developing own national ensemble flood forecasting capacity (e.g. UK, Switzerland, France, Sweden, Germany)

www.kcl.ac.uk

EFAS inspires innovation at national level

• EFAS proves ensemble flood forecasting can be done operationally – EFAS precedent often

referred to in R&D strategies

“Interviewer: Was the idea sort of influenced by EFAS or it had something else to do with it?

Forecaster: I have seen presentations by Jens Bartholmes and Jutta Thielen. This was an inspiration of course.”

www.kcl.ac.uk

Outline of Talk Today

1. Background to Study 2. General understandings of ensembles

and uncertainty in flood forecasting 3. Responses to EFAS in particular 4. Challenges to using EPS 5. Conclusions & recommendations

www.kcl.ac.uk

4. Challenges to using EPS

Displaying full ensemble may lead to: • confused inaction:

“Because we can't predict exactly . . . The problem is how to show this uncertainty. But this [EPS] is too much information, I think, that the public cannot use”

• false confidence and over-optimization of response:

“But even if you have a wide range, you can't be sure that really the water level is outside the range. I think this is really a problem. People think, ‘I'm safe, my threshold isn't reached.’ But the threshold can be reached, can be the higher water level...you can't be sure that the water level is in this range. Therefore the benefit [of EPS] is not very high”

www.kcl.ac.uk

Communicating Probabilistic Forecasts to Civil Protection

Forecaster doubts over ability of CPAs to understand probabilistic warnings: “But these people simply don’t understand. They don’t need this

information. ‘I don’t care what the probability is. Give me exact figure!!’ [they say]. It really doesn’t operate on uncertainties. I said, there is uncertainty of 10%. ‘What does it, what do you mean, 10% uncertainty? Give me the figure. I want exact forecast.’ [Laughs]”

“People cannot deal with uncertainties, it is too complicated. The problem is that

to live, to go for a walk, to know whether we go to the picnic or nor, we could cope with it. However, when it is time to decide whether we evacuate or not, it is another story”.

Training and better EPS visualizations can help, but ....

www.kcl.ac.uk

Conflicting institutional demands

Greater certainty “We believe that there must be a step change in the quality of flood warnings… The public and emergency responders must be able to rely on this information with greater certainty than last year” (Pitt 2008: vii)

And Earlier warnings

“The Met Office and the Environment Agency should issue warnings against a lower threshold of probability to increase preparation lead times” (Recommendation 34: Pitt review)

www.kcl.ac.uk

Accountability concerns over EPS

Re-distributing responsibility for flood risk management decisions

“EPS also means dumping responsibility onto forecast users. By forcing forecasters to provide deterministic predictions, the accountability remains entirely on the shoulders of forecasters. If a forecaster provides a probabilistic forecast, they give the import for the decision to forecasts users. ... Asking for a deterministic prediction is also a way for the person in charge of taking a decision to avoid decisional problems and blame.”

Concerns over Europeanization and competencies “If the [national river forecasting centre] starts talking about weather, they get

heavily criticised. If we mention the word flooding, ... we got heavily criticised...If you have a European Agency [EFAS] with an unclear mandate that does not have a national responsibility, providing such a service, then we would be very nervous about this.”

www.kcl.ac.uk

Outline of Talk Today

1. Background to Study 2. General understandings of ensembles

and uncertainty in flood forecasting 3. Responses to EFAS in particular 4. Challenges to using EPS 5. Conclusions & recommendations

www.kcl.ac.uk

Conclusions • Informational value of EPS not self-evident

– understood in different ways by operational flood forecasters

– Ensembles not always used as proponents imagine they should be

• Meaning and value of EFAS thresholds unclear • Substantial institutional challenges to ensemble

flood forecasting in general and to EFAS in particular

www.kcl.ac.uk

Recommendations for further improvements

• Improve calibration of thresholds • Tailor threshold levels to needs of different

end users • Clarify and communicate how thresholds

are calculated • Experiment with providing hydrographs to

some cooperating water authorities • Visit forecasting centres and provide more

training in situ