modelling loss data established practices and open issues giulio mignola intesa sanpaolo

Modelling Loss DataEstablished practices and Open Issues

Giulio MignolaIntesa Sanpaolo

2

Disclaimer

Opinions and views expressed here are of the speaker exclusively and do not necessarily represent practice or policies of Intesa Sanpaolo Spa

http://www.intesasanpaolo.com/

3

Summary

Basics of Op Risk Modelling - the LDA established practices– Fundamental elements– Essential Tools

Open issues in LDA– Which distribution to choose: a.k.a. “the extrapolation problem”– Point vs. Interval estimate: how big are the error bars?

Conclusions


4

Fundamental elements of OpRisk: the LDA

Total Loss Distribution

Op. VaR 99.9 percentile

Unexpected losses

0

2

4

6

8

10

12

14

1 3 5 7 9 11 13 15 17 19

# events per period

rela

tive

freq

uenc

y

Aggregated via Monte Carlo or analytical tools

Severity distribution Frequency distribution


5

Essential ToolsFrequency distributions

Poisson distribution:– Expected Value = Variance =

Negative Binomial distribution:– Expected Value = r ;– Variance = r (1+) > r

Histogram of frequency

n

Fre

quen

cy

0 10 20 30 40 50 60

0

50

100

150

poissonnegative binomial var/mu=3


6

Essential ToolsSeverity distributions

Lognormal lognormal-gamma mixture Generalised Pareto density Inverse Gaussian Weibull Burr density

same first two moments

x

1 -

CD

F

100 10,000 1,000,000 100,000,000 10,000,000,000

1 e-10

1 e-07

1 e-04

1 e-01

lognormalgpdinvGausslnorm-gammaweibullburr


7

Essential ToolsFitting distributions to data

Maximum Likelihood Estimation– Best asymptotic properties– Simple form, well established numerical methods– Sensitivity to outliers

Robust estimation methods– L-moments are more robust than ordinary sample moments– Trade-off between variance and stability– Attention to implementation complexity and introduction of new parameters– Example 1: Minimum Distance Estimator minimises a goodness-of-fit measure, like Cramér

von Mises (introduces no new parameter, simple form)– Example 2: Optimal Bias-Robust Estimator is an M-estimator with bound on Influence

Function and iterative estimation of most efficient weights (one new parameter [the bound], complex implementation [double minimisation])


8

Essential ToolsBuilding the annual loss distribution

Monte Carlo – old stuff very well known– Easy to implement– Easy to include insurance– Could be time and memory consuming – to achieve accuracy one may need millions of

scenarios, although some simple variance reduction techniques could be applied (e.g., Latin hypercube sampling)

– If dependence among dimensions is introduced, an ex-post aggregation mechanism has to be implemented

FFT - Using properties of the characteristic function to get the probability distribution of the aggregate loss

– Very fast– Not so straightforward to include insurance– One dimension at a time – ex post aggregation of dimensions is needed

Single Loss approximation – analytical approximation– The level of accuracy is not straightforward to keep under control– Insurance is structurally difficult to implement


9

A useful instrumentThe single loss “approximation”

Following known results recently highlighted in C. Klüppelberg and K Böcker, “Operational VaR: a Close-Form Approximation)

For a large class of severity distribution (sub-exponential) it can be proven that

And hence for a high percentile of the aggregated distribution the following approximation holds

nxFxF n

x

)()(

lim*

][E1111

npFpFS

))(1]([E)(1 xFnxFS


10

How it works in practiceAn example


11

From the hyperuranic of ideas to the cavern of implementationsi.e.what doesn’t work so well in the LDA


12

Understanding some challenges posed by Basel II

In the Basel Accord on Capital Adequacy Operational Risk is identified as a distinct source of banking risk and a specific capital requirement is introduced; in addition to basic and standard measurement methodologies, stringent requirements for the internal models are defined.

The crucial quantitative requirement characterizing the AMA is to produce a risk measure which is “compatible” with the 99.9% of the annual loss distribution (i.e. the distribution of the total loss of one year).

Another stringent requirement is the collection of loss data (both internal and system) that covers at least 5 years history

The BIS tower in Basel


13

Implications on models (1)

The value of the annual loss, assuming an average frequency of k losses per year, can be evaluated summing, on average, k losses independently coming from the severity distribution.

To describe the distribution of the annual loss the whole procedure should be repeated many times, at least 1000 times to reach the required 99.9 % confidence level. As a result k 1000 losses should be independently extracted from the loss severity distribution

The bank is supposed to collect each year on average k new losses then to reach the k 1000 losses in the database

This means that the amount of data required is equivalent to a 1000 bank years

………

999 annual losses1000th annual loss

999th annual loss

VaR = 99.9 % confidence level

=

Annual Loss k Individual losses

+ + ++ …

Loss DB 1000 = 1000 bank years

or k 1000 losses


14

Implications on models (2)

Operational risk loss distributions belong to the class of sub-exponential distributions. This implies that, in the tail, the aggregate annual loss is roughly equal to the largest single loss Consequently the properties of losses in the far tail of the severity distribution completely drive the VaR result.

The 99.9% annual loss confidence level with a 5 years loss history in the database, entails a 1000/5 = 200-fold extrapolation beyond the observed data. External data from the industry could help a bit, but at least a data pool of 200 banks for 5 years is needed. Homogeneity of the various contributors to the pool remains questionable.

These elements together imply that to obtain the result one must extrapolate. Extrapolation is very risky since it cannot be proven by observation: results perfectly fine and compatible in the realm of observed data could diverge dramatically by extrapolating. And the problem is unsolvable unless some other information is brought in. A fair amount of guesswork, isn’t it ?

…

1000 banks

or

Observed data

Extrapolated data

=

Annual Loss Individual losses

+ + ++ …

Single largest loss


15

What does it mean in practiceWhich distribution to choose

Since there are no established “dynamic” models of the loss generation, only phenomenological approaches based on collected loss data are viable.

The selection process has to be done relying on goodness of fit statistics (Kolmogorov - Smirnov, Anderson - Darling, Smirnov - Cramér - von Mises, Schwartz Bayesian Criterion, etc.)

There are cases in which two, or even more than two, equally good fits (statistically speaking) are found


16

An example

function name: dlnorms

estimated parameters:

meanlog 11.204300 +- 0.139508

sdlog 1.753589 +- 0.098629

constant parameters:

shift 50000.000000

neg. loglik. value: 2083.305

fitted range: ( ) 50000 Inf num. data fitted: 158

cdf extremes: 0.7768362 1

chisquare KS AD SCvM SBC

g.o.f. 1.999580149 0.09043227 0.001324199 0.0005123347 4176.734

p.val 0.005948592 0.04330166 0.347246805 0.2094978905 -4176.734

function name: dgpd


shape 0.799519 +- 0.151729

scale 86880.052984 +- 13819.608154


loc 50000.000000





g.o.f. 1.79569796 0.05435913 0.001259334 0.0003931439 4172.585

p.val 0.01779608 0.24924875 0.280444997 0.3187397029 -4172.585


17

… but far in the tail using the single loss approximation (N = 180)

Less 100 million for fit1and more than 500 million for fit2


18

Why it happens i.e. how bad the extrapolation problem can be

To reach the 99.9 percentile of the annual aggregate loss we should explore regions of the loss severity which are far beyond the observed data points

Example: if we want the 99.9 percentile of an aggregate loss obtained with a frequency of 100 we need to explore losses up to

or 1 in 100,000 losses. Recognizing that in a typical “risk class” we have in the most favourable condition few thousands of data points it is almost sure that we need to guess the behaviour of losses in the tail.

99999.0100

999.011

p


19

What can be doneExternal data could help

log(loss)

% p

oint

s of

qno

rm

0.1

1

10

50

90

99

99.999.9799.99

99.999

99.9999

internal data only

log(loss)

% p

oint

s of

qno

rm0.1

1

10

50

90

99

99.999.9799.99

99.999

99.9999

internal + external data

internal dataexternal data


20

What can be done… but for the time being is not the solution

log(loss)

% p

oint

s of

qno

rm

0.1

1

10

50

90

99

99.999.9799.99

99.999

99.9999

internal + external data

internal dataexternal datainternal dataexternal data


21

Typical consequences

Issues on model output Instability of results (also favoured by the “huge” statistical uncertainties coming from the fitting of

the severities), i.e. re-estimating the severities parameters could produce a change with respect to previous cycle results of material size (even greater than 100%)

Unreliability of results - uncertainty on the correct level of capital How to explain to top management (and regulators) that, given no relevant changes in the data,

in March you need 40% more capital than in December ?

Implications on model validation Absolute validation is unachievable Need to focus on trends of the results Needs to heavily rely on benchmarks

Implications on financial system consistency What really level playing field really means


22

Is there a solution ?

With the current formulation of the regulation, probably not

Few simple ideas on how to proceed Internal validation should be always performed jointly with an external party (to provide enough

experiences to benchmark model outputs). Potentially we could see a new figure emerging here: “The model validator”

A mechanism to incorporate trends (taking as a basis to measure variation an in-sample anchor point) should be hardwired in the model (even at the price of a potential inconsistent description of sample data)

Taking a more radical attitude Promoting Basel III, from now, and select a lower level of confidence, in, or very near, the

observed data range, then find another system to reach the regulatory comfort zone, e.g. by fixing a system multiplier


23

Other sources of unceratinty

Statistical uncertainty– Better to consider it carefully– Don’t rely on VaR calculated on point estimates of the severity parameters

Numerical approximations– Monte Carlo– FFT– Aggregation of different risk types

• Copulae vs. naïve approach• Which measure for dependency (rank vs. linear correlation)• Which data to use (severity, frequency, aggregated losses …)


24

Point vs. interval estimate: Statistical uncertainty

The result of a fit is a set of parameters together with the associated uncertainty

function name: dlnorms


meanlog 11.204300 +- 0.139508

sdlog 1.753589 +- 0.098629


shift 50000.000000





g.o.f. 1.999580149 0.09043227 0.001324199 0.0005123347 4176.734

p.val 0.005948592 0.04330166 0.347246805 0.2094978905 -4176.734

)var()var(22

PPP Neglecting correlations between parameters


25

Lognormal(meanlog=10,sdlog=2)

0.00 0.05 0.10 0.15 0.20

0.0

0.5

1.0

1.5

rel error on sdlog

rel e

rror

on

quan

tile

freq=200freq=1000

P/P, where P is the 99.9 percentile of the aggregated loss distribution:-severity: lognormal(10,2)-frequency: Poisson(200), Poisson(1000)

These results show how the statistical uncertainty on the estimated parameters reflects on the value of the VaR. Note the size of the error up to 70%. In the hypothesis of a symmetric function, the meaning of the statistical error is that another sample of the same size independently drawn from the same underlying distribution could be, with 68% probability, in between –70% and + 70% from the true value


26

A recent development:Correctly computing VaR with uncertainty

As was recently pointed out in X. Luo, P.V. Shevchenko and J.B. Donnely, the skewness in the response of the quantiles with respect of parameters variation should be correctly taken into account since naively ignoring this effect could grossly underestimate the risk figure.

A worked example Lognormal severity (a typical situation) Severity parameters (0, 0) and their uncertainty (, ) Is VaR0 = VaR(0, 0) – that is the case in which uncertainty on parameter estimates is not

considered – the good risk measure to be used ? The answer is definitely No, the correct measure should be:

– VaR = E[VaR(, )] over the , distributions which properly take into account their uncertainty. With a good degree of precision both distributions could be considered normal centred in (0, 0) with standard deviations (, )


27

A practical example

1.6 1.8 2.0 2.2 2.4

0

1

2

3

4

sdlog

dens

ity(s

dlog

)

Uncertainty on sigma Uncertainty on VaR

Expected sigma = sigma0 Expected VaR ≠VaR0

40% ΔVaR10%Δσ

0e+00 1e+08 2e+08 3e+08 4e+08 5e+08 6e+08

0e+00

1e-09

2e-09

3e-09

4e-09

5e-09

6e-09

VaR

dens

ity(V

aR)

VaR0 = 111 ME[VaR] = 160 M


28

A full MC is needed if parameters space is multidimensional

VaR

Freq

uenc

y

0e+00 1e+08 2e+08 3e+08 4e+08 5e+08 6e+08

0

5

10

15VaR0=111 ME[VaR]=139 M

%2,%5

%25 VaR


29

Conclusions

The financial industry has reached a high level of understanding of the Operational Risk discipline and a rather sophisticated level of modelling the risk profile

– Approaches to modelling operational risk losses have become well engineered with the use of smart statistical techniques

– Methods have been developed (in some cases intelligently borrowed from other disciplines) to solve difficult problems

– Software tools are available to “industrialize” the VaR production

… but we are still facing some “unpleasant” effects which are difficult to properly understand and to keep under control. In particular an unlucky definition of the risk measure imposed by the prudential regulation (Basel II) is fundamentally flawing the soundness of the measurement framework. This has to:

– be understood in the first place– worked around, even at the price of loosing some theoretical consistency– trigger an initiative of lobbying to properly change the current formulation of the regulation.

In order to bring back to a manageable situation the Operational Risk measurement framework giving back a proper meaning to the op risk measures, the validation of models and, last but not least, the level playing field we all claim to want in place, the regulation should be improved.


30

References

G. Mignola and R. Ugoccioni, Tests of Extreme Value Theory, Operational Risk, Vol. 6, Issue 10, Oct. 2005.

G. Mignola and R. Ugoccioni, Sources of uncertainty in modelling operational risk losses, Journal of Operational Risk Vol. 1, No. 2, Summer 2006.

G. Mignola and R. Ugoccioni, Effect of a data collection threshold in the loss distribution approach, Journal of Operational Risk Vol. 1, No. 4, Winter 2006/07.

X. Luo, P.V. Shevchenko and J.B. Donnely, Addressing the impact of data truncation and parameter uncertainty on operational risk estimate, Operational Risk, Vol. 2, Number 4, winter 07/08.

S.A. Klugman, H.H. Panjer and G.E. Willmot, Loss Models: from Data to Decisions, 2nd ed., John Wiley and Sons, New York, 2004.

P. Embrechts, A.J. McNeil and R. Frey, Quantitative Risk Management, Princeton University Press, Princeton, 2005.

J. Shih, A. Samad-Khan and P. Medapa, Is the size of an Operational Loss Related to Firm Size, Operational Risk (January 2000); A. Samad-Khan An integrated Framework for Measuring and Managing Operational Risk, OpRisk 2003.

S. Carrillo Menendez and A. Suarez, Version 2.0 Capital Models for Operational Risk: Bringing Economic Sense to Economic Capital, (talk at Global Conference on Operational Risk, New York, March 13-17, 2007)

F. Beekman, Using Robust Estimators to Find Parameters, (presentation given at Global Conference on Operational Risk, New York, March 13-17, 2007)


modelling loss data established practices and open issues giulio mignola intesa sanpaolo

Documents