Assessment of flood predictive

Department of Earth and Geo-Environmental Sciences University of Bologna

Assessment of flood predictive uncertainty in using meteorological ensembles to hydrological forecasting

Prof. Ezio Todini

ISSAOS 2005, L’Aquila, August 29 – September 2, 2005

Department of Earth and Geo-Environmental Sciences University of Bologna

The basic problem Flood emergency managers require simple decision rules to be applied in real time.

Unfortunately these rules are strongly affected by the uncertainty on the future evolution of events.

This implies the necessity of using forecasting tools with the aim of reducing their uncertainty on future events.

Department of Earth and Geo-Environmental Sciences University of Bologna

Do we all agree on what is a Flood Forecasting System?

A Flood Forecasting System is a tool aimed at reducing uncertainty on the future evolution of a flood event..!!!

It is the reduction of uncertainty that allows for more reliable decisions.

Department of Earth and Geo-Environmental Sciences University of Bologna

Level

An Example:The Flood Warning Problem Probability of overflow

Expected value = Forecast

Costs

Cross section

Department of Earth and Geo-Environmental Sciences University of Bologna

The big question: are we actually describing it?

When using such forecasting tools, as pointed out

by Krzysztofowicz (1999), it is essential to asses the uncertainty of the future state of the quantity of interest (level, discharge, etc.) conditional to its forecasted value in order to improve decisions.

This approach is a much simpler alternative to the overwhelming computational effort required by a full unconditional uncertainty assessment, which would imply marginalisation of the forecasting density with respect to all possible forecasting models, model parameters, initial states, input measurement and forecasting errors.

Department of Earth and Geo-Environmental Sciences University of Bologna

The full unconditional uncertainty assessment f ( y ) = ∫∫∫∫∫ f (y, Xˆ ,ϑ, I , M , X ) dX dM dI dϑ dXˆ

= ∫∫∫∫∫ f ( y Xˆ ,ϑ, I , M , X ) f (ϑ I , M , X ) f ( I M , X ) f ( M ) f ( X ) f (Xˆ )dX dM dI dϑ dXˆ

with y=

the quantity of interest, the predictand

X= Xˆ =

the input data

M= ϑ=

the model

I=

the forecasted inputs

the model parameters the initial status

Department of Earth and Geo-Environmental Sciences University of Bologna

Different levels of conditional uncertainty 1/5 The full conditional density

f ( y Xˆ , ϑ , I , M , X )

expresses our uncertainty conditional to the forecast provided by a given model with given parameter values, given initial status, given inputs and given input forecasts.

Although incorrect, it is quite common to estimate the model parameter values ϑˆ and to use the conditional density f y Xˆ , ϑˆ, I , M , X to represent

(

our predictive uncertainty.

)

Department of Earth and Geo-Environmental Sciences University of Bologna

Different levels of conditional uncertainty 2/5 A more appropriate way is to derive a “posterior density” of the parameters using Bayesian Inference f " (ϑ Y , I , M , X ) using Y a set of past observations and to marginalise with respect to the parameters, to obtain:

f ( y Xˆ , Y , I , M , X ) = ∫ f ( y Xˆ , ϑ , I , M , X ) f " (ϑ Y , I , M , X ) dϑ

Department of Earth and Geo-Environmental Sciences University of Bologna

Different levels of conditional uncertainty 3/5 If a density representing the uncertainty of the forecasted inputs is available,then one can obtain: f ( y Y, I, M, X ) = ∫

[∫ f (y Xˆ ,ϑ, I , M , X ) f (Xˆ )dXˆ ] f "(ϑ Y , I , M , X ) dϑ

Department of Earth and Geo-Environmental Sciences University of Bologna

Different levels of conditional uncertainty 4/5 NOTE that, the previous slides show that no action is generally taken to eliminate the conditioning -

on the assumed model structure,

-

on the initial value of the state variables,

-

on the input measurement errors.

Department of Earth and Geo-Environmental Sciences University of Bologna

Different levels of conditional uncertainty 5/5 As a matter of fact the model is generally taken as our prior knowledge, concentrating all the uncertainty in its parameter values.

The initial value of the state variables heavily affects the predictions, but its effect can be reduced using continuous-time models and starting at the end of dry periods.

Input measurement errors will certainly affect parameter estimation, but if they are not eccessive, they will not strongly affect the predictive uncertainty.

Department of Earth and Geo-Environmental Sciences University of Bologna

ASSESSING PARAMETER ESTIMATION ERROR In order to overcome the problem of formulating “formal” likelihoods, correctly representing the statistical properties of the error terms Beven and Binley, (1992) introduced the Generalised Likelihood Uncertainty Estimation (GLUE), which follows in principle the Bayesian inference scheme, but uses “less formal likelihoods” as defined by the authors.

Department of Earth and Geo-Environmental Sciences University of Bologna

The introduction of less formal likelihood functions overcomes the need for formulating precise distribution functions for the observable variables and/or for the errors in complex situations originated by the presence of many sources of errors, the complexity of the explicative models considered and the high number of parameters with which to build the learning process.

Unfortunately, as it was shown by Mantovan and Todini (paper in preparation), the use of less formal likelihood functions leads to paradoxical inferential results.

Department of Earth and Geo-Environmental Sciences University of Bologna

Simulation results using three well known GLUE “less formal likelihoods” ⎧⎡ s 2 (ϑ ) ⎤ N ⎪⎢1 − n 2 ⎥ 1 Ln (ϑ ) = ⎨⎣ sn ⎦ ⎪ ⎩0

[

sn2 (ϑ ) ≤ sn2 sn2 (ϑ ) > sn2

]

L2n (ϑ ) = sn2 (ϑ )

[

−N

N ≥1

]

L3n (ϑ ) = Exp − Nsn2 (ϑ )

where

N ≥1

sn2 (ϑ ) = Var [ yn − yˆ n (ϑ , X n )] sn2 = Var [ yn ]

Nash-Suttcliffe Inverse error variance Exponential

Department of Earth and Geo-Environmental Sciences University of Bologna

Synthetic data testing of GLUE A two dimensional first order autoregressive process based on two parameters, which known true value was set to (Ď‘1* ,Ď‘2* ) = (1,1) was generated with the addition of a non central Student-t distributed autoregressive noise. The samples were of length n=m=336.

The results using the three non formal likelihoods are compared with the ones obtained using the Gaussian (wrong, but in this case relatively robust) assumption

Department of Earth and Geo-Environmental Sciences University of Bologna

Bayesian Inference: Gaussian Likelihood

Department of Earth and Geo-Environmental Sciences University of Bologna

GLUE: Nash-Suttcliffe Likelihood

Department of Earth and Geo-Environmental Sciences University of Bologna

GLUE: Inverse Error Variance Likelihood

Department of Earth and Geo-Environmental Sciences University of Bologna

GLUE: Exponential Likelihood

Department of Earth and Geo-Environmental Sciences University of Bologna

Bayesian Inference: true likelihood (Posterior to Prior distance)

Department of Earth and Geo-Environmental Sciences University of Bologna

Less formal likelihoods (Posterior to Prior distance) Figure 5. Distance between posterior and prior p.f.: Nash-Sutcliffe l.f.l..

0.04

0.035

0.03

Frobenius norm

0.025

0.02

0.015

0.01

0.005 5-th percentile 50-th percentile 95-th percentile 0

0

2

4

6

8

10 Sample size (weeks)

12

14

16

18

20

Department of Earth and Geo-Environmental Sciences University of Bologna

Bayesian Inference: true likelihood (Posterior expected square error loss) Figure 8. Posterior expected squared error loss: Exact likelihood.

0.12 5-th percentile 50-th percentile 95-th percentile

0.1

Frobenius norm

0.08

0.06

0.04

0.02

0

0

2

4

6

8

10 Sample size (weeks)

12

14

16

18

20

Department of Earth and Geo-Environmental Sciences University of Bologna

Less formal likelihoods (Posterior expected square error loss)

Department of Earth and Geo-Environmental Sciences University of Bologna

Similar results were found in the Po river case study (37,000 km2 ) by comparing:

- GLUE using a less formal likelihood based on Nash-Sutcliffe statistics

- Bayesian inference assuming a χ2 density for the sum of squared errors

Department of Earth and Geo-Environmental Sciences University of Bologna

GLUE: Nash-Sutcliffe

Department of Earth and Geo-Environmental Sciences University of Bologna

GLUE: Nash-Sutcliffe

Department of Earth and Geo-Environmental Sciences University of Bologna

Bayesian Inference:

χ2

Department of Earth and Geo-Environmental Sciences University of Bologna

Bayesian Inference:

χ2

Department of Earth and Geo-Environmental Sciences University of Bologna

GLUE: Nash-Sutcliffe (4 x 400)

Department of Earth and Geo-Environmental Sciences University of Bologna

GLUE: Nash-Sutcliffe (2 x 800)

Department of Earth and Geo-Environmental Sciences University of Bologna

GLUE: Nash-Sutcliffe (1 x 1600)

Department of Earth and Geo-Environmental Sciences University of Bologna

Bayesian Inference:

χ2

(4 x 400)

Department of Earth and Geo-Environmental Sciences University of Bologna

Bayesian Inference:

χ2

(2 x 800)

Department of Earth and Geo-Environmental Sciences University of Bologna

Bayesian Inference:

χ2

(1 x 1600)

Department of Earth and Geo-Environmental Sciences University of Bologna

Another example using GLUE on a Chinese 10,000 km2 catchment

Department of Earth and Geo-Environmental Sciences University of Bologna

The Bayesian inference using the Normal Quantile Transform on the same Chinese catchment

Department of Earth and Geo-Environmental Sciences University of Bologna

Inequifinality as opposed to equifinality If we are able of determining a relatively peaky posterior density for the parameters we can resonably question ourselves whether the use of a set of parameters can strongly distort the estimation of the model predictive density. In other words should we use the conditional f (y Xˆ ,ϑˆ, I , M , X ) or the unconcitional density? f ( y Xˆ , Y , I , M , X ) = ∫ f ( y Xˆ , ϑ , I , M , X ) f " (ϑ Y , I , M , X ) dϑ

Department of Earth and Geo-Environmental Sciences University of Bologna

Predictive uncertainty in hindcast model (A Chinese catchment) Unconditional

Conditional As one can see from this slide and from the following one, the difference between unconditional and conditional densities can be relatively small if the parameter posterior density is dense around the modal value.

Department of Earth and Geo-Environmental Sciences University of Bologna

Predictive uncertainty in hindcast mode (A Chinese catchment) Solid: Unconditional Dashed: Conditional

Department of Earth and Geo-Environmental Sciences University of Bologna

Predictive uncertainty in forecast mode In order to provide the probability density of the future values of our predictand (stage, discharge, etc.) conditional on our forecasts, we need to answer the following questions: Which are the means we have to express the input (mostly future rainfall) predictive uncertainty?

How do we operate in practice ?

Department of Earth and Geo-Environmental Sciences University of Bologna

Presently, we can hardly make direct use of the members of the Ensemble Forecasts provided by the Numerical Weather Prediction models.

For instance, the following example on the Po river in Italy shows that while the “deteministic” run (dashed line) provides a relatively good rainfall forecast, the “spaghetti” ensemble produces extremely biased estimates (possibly due to the coarser model mesh).

Similar results obtained in project EFFS using different NWP models confirm this situation.

Department of Earth and Geo-Environmental Sciences University of Bologna

At NOAA a post-processor for rainfall products generated by NWP models has been developed using the NQT and the ensemble mean (Slide: John Schaake).

Conditional Distribution

Joint distribution Forecast ZY

No assumption of normality for observed & forecast distributions

Inverse NQT For a given forecast 0

ZX

Archived data NQT

X

P(ZX zX0 | ZY = zY) zX0

Observed

ZX

NQT = Normal Quantile Transform zX0

Observed

PQPF given a QPF

Observed

zY0

Y

0

Normal Space

1

Normal Space

Department of Earth and Geo-Environmental Sciences University of Bologna

Which allows to correctly reproduce the future precipitation distribution (Slide: John Schaake).

Ensemble Member Climatologies 1

Probability

0,8

0,6

Obs Raw Syn

0,4

0,2

0 0

10

20 Daily Precipitation (mm)

30

40

f (X Xˆ )

Department of Earth and Geo-Environmental Sciences University of Bologna

John Shaake has shown that using the NQT and the ensemble mean it is possible to derive a probability density of future rainfall conditional to the ensemble mean, namely f (X Xˆ ) In real time, one can then derive one conditional density for each member of the ensemble and marginalise out the uncertainty caused by the precipitation forecasts.

Department of Earth and Geo-Environmental Sciences University of Bologna

Conversion of data in the Gaussian space

Gaussian Space

Normal Quantile Transform

Original Space

The Normal Quantile Transform (NQT) Both measured and modeled values are transformed into a Standard Normal space by quantile matching.

In the Normal space the derivation of the conditional densities is relatively simple. The conditional densities are Then reconverted back into the original space.

Department of Earth and Geo-Environmental Sciences University of Bologna

Conversion of data in the Gaussian space The NQT fully preserves the rank correlation. Moreover in the Gaussian space, the joint probability distribution of the observed and modelled variables is a multi-dimensional Gaussian distribution which density can be easily estimated.

In hindcast mode one can directly derive the predictive density, while in forecasting mode the predictive density can be derived following the work of Krzysztofowicz (1999).

Department of Earth and Geo-Environmental Sciences University of Bologna

The conditional density in hindcast mode

y

η

5

y = 0,7403x 2

R = 0,4958 4 3

f (ηi ηˆi )

2 1 0 -4

-3

-2

-1

0

1

2

3

4

-1 -2 -3

ηˆ

-4

yˆ i (ϑk , x i )

yˆ

Department of Earth and Geo-Environmental Sciences University of Bologna

If we accept the conditionality on the chosen model, on the initial state and on the measurement errors, the problem of predictive uncertainty, can be solved by marginalising the joint density with respect to the parameters and to the forecasted input.

Therefore, in practice the original equation f ( y Y, I, M, X ) = ∫

[∫ f (y Xˆ ,ϑ, I , M , X ) f (Xˆ )dXˆ ] f "(ϑ Y , I , M , X ) dϑ

must be discretised to be used with an ensemble of parameter values drawn from the parameter space and an ensemble of rainfall forecasts produced by the NWP models.

ϑi

∀ i = 1, N s

Department of Earth and Geo-Environmental Sciences University of Bologna

One possible approach can thus be summarised as follows: 1) Assume a multi-uniform prior f ' (ϑ ) on the parameters; 2) Generate, a large number Ns of parameter sets ϑi ∀ i = 1, N s by means of a Monte Carlo approach; 3) Estimate the posterior density of the parameters f " (ϑ Y , I , M , X ) using the Bayesian Inference process on historical records; 4) Estimate, for each generated parameter set, its posterior probability of occurrence f " (ϑi Y , I , M , X ) ;

Ne

Department of Earth and Geo-Environmental Sciences University of Bologna

5) For each ensemble member and each parameter set estimate the predictive density f (y Xˆ j ,ϑi , I , M , X ); 6) Marginalise out the ensemble uncertainty by uniformly weighting the ensembles f ( y ϑ , I , M , X ) = 1 ∑ f (y Xˆ ,ϑ , I , M , X,) with Ne

i

Ne

Ne

j =1

j

i

the number of ensemble member used;

7) Marginalise out the parameter uncertainty: Ns

f ( y Y , I , M , X ) = ∑ f ( y ϑi , I , M , X ) f " (ϑi Y , I , M , X ) i =1

Department of Earth and Geo-Environmental Sciences University of Bologna

Possible alternatives One has to realise that the required computational effort can be quite substantial given that Ne ,the number of ensemble members is generally around 50 and Ns the number of parameter sets is generally of the order of several thousands (t10000).

Therefore, given that one must repeat the procedure for each future value, two alternatives can be conceived, that can reduce the computational effort as well as the required computation time.

Department of Earth and Geo-Environmental Sciences University of Bologna

Alternative 1 The first alternative, similarly to what is done in meteorology, is to cluster the parameter sets in groups (possibly into equiprobable groups) and use a representative set per each group. The number of these groups could be of the order of 100, thus reducing by two orders of magnitude the computational burden.

Department of Earth and Geo-Environmental Sciences University of Bologna

Alternative 2 If one can prove the substantial coincidence of the predictive density marginalised with respect to the parameters and of the one conditional to an estimated parameter set value (for instance a ML estimated value) as in the Chinese catchment case, it is possible to further reduce the computational effort by repeting steps 5 and 6 as follows.

Ne

Department of Earth and Geo-Environmental Sciences University of Bologna

Alternative 2 5) For each ensemble member estimate the predictive density f (y Xˆ j ,ϑˆ ∗ , I , M , X ) ; 6) Marginalise out the ensemble uncertainty by uniformly weighting the ensembles f (y ϑˆ , I , M , X ) = N1 ∑ f (y Xˆ ,ϑˆ , I , M , X ) , with ∗

Ne

e j =1

Ne

∗

j

the number of ensemble member used;

Department of Earth and Geo-Environmental Sciences University of Bologna

CONCLUDING REMARKS The objective of this lecture was not to propose solutions, rather to set forth the perception of the main unresolved problems in the assessment of Real Time Flood Forecasting Uncertainty and to propose and discuss a number of possible alternative approaches.

It must be clear that these are ideas and research lines following which we hope to find the appropriate solutions within the frame of HEPEX and of a number of European funded research projects.