Model Risk & Model

ValidationRajib Chakravorty

The views expressed in this presentation are that of my own and they

do not represent anything of my current assignment or my current employer

Agenda

What is Model & Model Risk Management

Modelling Process

Sources of Model Risk

Regulatory Timelines

What is Model Validation

Model Validation Example

Guidelines for Model validation

Summary

The world we live in is vastly different from the world we

think we live in.

Nassim Nicholas Taleb

We know from chaos theory that even if you had a perfect

model of the world, you'd need infinite precision in order

to predict future events. With socio-political or economic

phenomena, we don't have anything like that.

Nassim Nicholas Taleb

In the risk management and compliance space, Taleb

argued that our corporations, industries, and economies

have become very fragile – a breeding ground for a Black

Swan event to occur and to have devastating and lasting

impact.

While statistical risk and pricing models may do a good

job when markets are calm, they lay the seeds for their

own destruction – it is inevitable that such models be

proven wrong. The riskometer is a myth (Danielsson

2009).

What is a Model?“The term model refers to a quantitative method, system, or

approach that applies statistical, economic, financial, or

mathematical theories, techniques, and assumptions to process

input data into quantitative estimates. A model consists of three

components:

an information input component, which delivers assumptions and data to

the model;

a processing component, which transforms inputs into estimates;

and a reporting component, which translates the estimates into useful

business information.” *

*Source : SR Letter 11-7 Attachment

Board of Governors of the Federal Reserve System Office of the Comptroller of the Currency

Examples of Models

Most organisations begin their inventory of models with the identification of model “classes”

aligned with business activities where model-based decision making occurs and which therefore

are potential sources of model risk.

Examples of model classes:

►Credit Risk (e.g., PD, LGD, Underwriting, Behavioral PD, Exposure)

►Treasury (e.g., ALM, liquidity risk)

►Stress Testing (e.g., credit loss forecasting, PPNR)

►Market Risk (e.g. Value-at-Risk)

►Trading Counterparty Credit Risk

►Operational Risk (e.g. Basel II models)

►Asset management (e.g. Portfolio Optimisation)

►Economic Capital

Model Risk Management

“The expanding use of models in all aspects of banking reflects the

extent to which models can improve business decisions, but models

also come with costs. There is the direct cost of devoting resources to

develop and implement models properly. There are also the

potential indirect costs of relying on models, such as the possible

adverse consequences (including financial loss) of decisions based on

models that are incorrect or misused.

Those consequences should be addressed by active management of

model risk.”

SR Letter 11-7

Attachment

Board of Governors of the Federal Reserve System

Office of the Comptroller of the Currency

Example of Model Failure

Statistical models that attempt to predict equity market

prices based on historical data. So far, no such model is

considered to consistently make correct predictions over

the long term. One particularly memorable failure is that

of Long Term Capital Management, a fund that hired

highly qualified analysts, including a Nobel Prize winner

in economics, to develop a sophisticated statistical

model that predicted the price spreads between

different securities. The models produced impressive

profits until a spectacular debacle that caused the then

Federal Reserve chairman Alan Greenspan to step in to

broker a rescue plan by the Wall Street broker dealers in

order to prevent a meltdown of the bond market.

-Get Representative Sample – will be

used to identify Analytic Solution to the

Business Problem

- Apply Global Exclusion Criterion

- Perform Data Integrity Checks

- Agree on Operational Definitions of

Dependent Attribute

- Create Development / Validation

Samples*

SupplierInputProcessOutputCustomer

Client Identification of an

Analytic Solution

Algorithm / Code to

Implement the

Analytic Solution

- Representative Sample at the required level –Account, Transaction, Relationship etc.

- Behavioral Information like: Response, Profit, Conversion, Lapsation, Quote etc.

- Operational Definition of above attributes.

- Overlay of External Database Information

Client

Start

Data Preparation

- Perform Univariate / Bivariate Analysis

- Cap Outliers

- Treat Coded Values Appropriately

- Missing Treatment (Mean / Median /

Regression Based Approach etc)

- Convert Character Attributes to

Numeric Attributes - Dummy Creation /

Ordinal Values etc.

-Multicollinearity Removal – Identify set

of attributes with no serious

multicollinearity

Build Model – Identify the Best Set

of Significant (>95%) Attributes that

explain the phenomena:

- Logistic track: Concordance / AIC

/ HL Goodness of fit / KS/

Parameter Significance / VIF / CI /

Odds Ratio / Rank Ordering

- OLS track: R Square (adj) /

Parameter Significance / VIF / CI /

Rank Ordering

Validate Model

- Refit the model on Validation Sample

- Score the model on Validation / Full

Population.

- Check Signs with Bivariate Analysis.

Implementation Strategy

-Identify the Implementation

Strategy.

- Create a Code to implement it

at Customers Database.

- Test and Implement

-Control Strategy

- Finalize a Tracking

/ Control MechanismEnd Model Validates

on all

parameters

Y

N

Modeling - High Level Process Map

Sources of Model Risk In Model Life-Cycle

Start ModelScoping

Data Preparation

Independent Data

Validation & Review

Model Development

Independent Model

Validation & Review

Model Approval

Implementation& Deployment

On-going Monitoring

& Periodic Re-Validation

Incomplete or

inaccurate model

development

data

Inconsistent

model set-up /

assumptions

Inconsistent

with business

objectives

Flawed model

theory,

approach, or

assumptions

Identifying and

choosing factors

that can be used to

estimate risks

Data

problems and

selection bias

Changes in

market

conditions

Model

Life

Cycle

Risk – Economic Capital Modeling

4. Integrate Risks to

Compute Firm’s

Economic Value

Distribution

3. Assess Dependencies

Among Risks

• Determine the Correlations

1. Identify Risks

2. Determine Firm’s Exposure to

Each Risk Driver

• Build Economic Value

Distributions

Risk

A. CREDIT B. INTEREST C. OTHER

MULTIPLE RISK DEPENDENCIES

1 in a 100

Expected Value

Pro

bab

ilit

y

50%

1%

Economic Value

D. INTEGRATION

Credit Risk

•Data Reconciliation, Obligor Research, Running Credit Tools (KMV/ CreditMetrics)

Interest Rate Risk

•Scenario generation, In-house models for term structure of interest rates, Duration-

Convexity analysis

Data – Fit for Purpose ?

Is the data suitable for the modelling task?

Reliability in data collection; eg how reliable is a self-

assessment of income?

Or, data based on an existing portfolio of predominantly

older customers is used to build a model for a card

targeting young customers.

A data set of accepted loan applications, to build a

scorecard across all new applications.

Market prices reflect the value of assets at any given time, but

that does not mean they provide a good signal of the state of

the economy or are a good input into forecast models. The

reason is that market prices reflect the constraints facing

market participants.

Robustness of data

There are some problem domains where risk factors and

distributions on variables are stable over time.

However, consumer credit does not remain stable over time.

Credit risk changes over the business cycle.

Credit usage behaviour changes over time.

Banks’ risk appetite changes over time.

Innovations in technology and product development change

risk.

All of these time-varying factors affect the applicability of

credit risk models over time.

Possible fundamental limitations

of predictive model based on data

History cannot always predict future: using relations derived from historical data to predict the future implicitly assumes there are certain steady-state conditions or constants in the complex system. This is almost always wrong when the system involves people.

The issue of unknown unknowns: in all data collection, the collector first defines the set of variables for which data is collected. However, no matter how extensive the collector considers his selection of the variables, there is always the possibility of new variables that have not been considered or even defined, yet critical to the outcome.

Self-defeat of an algorithm: after an algorithm becomes an accepted standard of measurement, it can be taken advantage of by people who understand the algorithm and have the incentive to fool or manipulate the outcome. This is what happened to the CDO rating. The CDO dealers actively fulfilled the rating agencies input to reach an AAA or super-AAA on the CDO they are issuing by cleverly manipulating variables that were "unknown" to the rating agencies' "sophisticated" models.

Source : https://en.wikipedia.org/wiki/Predictive_modelling

What is Model Validation for Basel

Compliance ?

Model validation is the set of processes and activities intended

to verify that models are performing as expected, in line with

their design objectives and business uses. Effective validation

helps ensure that models are sound. It also identifies potential

limitations and assumptions and assesses their possible impact.

“Validation…requires verification of the minimum requirements

for the IRB approach.”*

“Validation should focus on…oversight and control procedures

that are in place…prompt reassessment of the IRB parameters

when the actual outcomes diverge materially from expected

results.”***Source: BCBS WP No.14 – Feb 2005

**Source: BCBS NL No.4 – Jan 2005

Validation Approach – Rating System

Internal Validation by

Individual Bank

Validation of Rating

System

Validation of Rating

Process

BenchmarkingBacktesting

Data

Quality

PD

Report Problem

& Handling

Internal Use by

Credit Officers

Risk

Components

Model

Design

LGD EAD

Broad Approach

to Validation

Source: BCBS Working Paper No. 14– Feb 2005

Total Population Base

(N obs)

Development Sample

(n1 obs)

Validation Sample

(n2 obs)

Validation could be done in 2 ways:

Validation Re-run

Scoring the Validation sample

Rerun the model on the validation sample.

Check the chi-sq values and level of significances

and p-values for each explanatory variable.

The p-values should not change significantly from

the development sample to the validation sample.

Check the signs of the parameter estimates. They

should not change from development sample to the

validation sample.

Check rank ordering. Both Development and

validation samples should rank order.

Validation sample scoringValidation Re-run

Score the validation sample using the parameter

estimates obtained from the scorecard developed on

the development sample.

Check rank ordering. Both development and

validation samples should rank order.

Scorecard Validation - Example

Lorenz Curve, Joint Lorenz, Gini Coefficient

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Random Development

Lorenz Curve

Lorenz curve indicates the lift provided by the

model over random selection.

Gini coefficient represents the area covered under

the Lorenz curve. A good model would have a Gini

coefficient between 0.2 - 0.35

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Random Development Validation

Joint Lorenz curve compares the lift provided by

the development sample and Validation sample.

For a stable and robust scorecard – the Lorenz

curves should be overlapping and similar in

distribution.

Approx. Bad Good Bad Overall Info Weight of Prob. Chi- % of % of Cum. % Cum. % % of Cum. % Gini Calc K-S score %-ile (B) (G) Rate Odds Odds Evidence Bad square all Bad allGood Bad Good obs. obs. Calcs

~10% 3 915 0.3% 0.0 0.1 (2.7) 0.00 37.3 0.7% 10.4% 0.7% 10.4% 10% 10% 0.00 9.7% ~20% 6 912 0.7% 0.0 0.1 (2.0) 0.01 31.7 1.4% 10.4% 2.2% 20.9% 10% 20% 0.00 18.7% ~30% 7 910 0.8% 0.0 0.2 (1.8) 0.01 29.9 1.7% 10.4% 3.9% 31.2% 10% 30% 0.00 27.4% ~40% 13 905 1.4% 0.0 0.3 (1.2) 0.01 20.4 3.1% 10.3% 7.0% 41.6% 10% 40% 0.01 34.6% ~50% 19 898 2.1% 0.0 0.4 (0.8) 0.02 12.7 4.6% 10.2% 11.6% 51.8% 10% 50% 0.02 40.2% ~60% 30 888 3.3% 0.0 0.7 (0.3) 0.03 3.3 7.2% 10.1% 18.8% 61.9% 10% 60% 0.04 43.1% ~70% 30 888 3.3% 0.0 0.7 (0.3) 0.03 3.3 7.2% 10.1% 26.1% 72.1% 10% 70% 0.05 46.0% ~80% 61 856 6.7% 0.1 1.5 0.4 0.07 9.8 14.7% 9.8% 40.8% 81.9% 10% 80% 0.11 41.0% ~90% 78 840 8.5% 0.1 2.0 0.7 0.08 33.8 18.8% 9.6% 59.7% 91.4% 10% 90% 0.16 31.8% ~100% 167 750 18.2% 0.2 4.7 1.6 0.18 399.5 40.3% 8.6% 100.0% 100.0% 10% 100% 0.39 0.0%

Totals 414 8,762 4.5% 0.05 1.0 581.70 0.79 46.0% B G B/G B/(B+G) Gini coefficient = 0.2925

Gini Coefficient and KS Measure of the Model: Development Data set

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Random Development

Logistic Solution – KS, Gini & Lorenz

Guiding Principles for IRB Validation of Models

Ensure integrity of IRB processes & systems

Confirm predictiveness of PD, LGD, EAD

All IRB components

Models

Inputs (Data) & Outputs (Estimates)

Rating Process

Control & Oversight Mechanisms (e.g., Internal Audit, Use)

Independent validation team

Qualitative and Quantitative techniques

Review of documents

Meet with various depts. (e.g., risk mgmt, audit, etc.)

Determine model type & rating philosophy

Check logic behind model (programs)

Review sample data

Benchmarking (i.e., compare w/ external sources)

Backtesting (i.e., estimates v. actual)

Regular and Periodic Basis

At least once per year

Changes in model, data or portfolio

Initial model development

TIMING (WHEN)

PURPOSE (WHY)

SCOPE (WHAT)

MEMBERS (WHO)

METHOD (HOW)

Summary

The models used in the bank do carry a risk which need to be identified and reported along with the mitigation plan or metrics

The quantification of ‘Model Risk’ is still not very clearly articulated and policies vary widely among banks

Model development and validation process need to be standardised across the bank with independent audit mechanism wherever possible.

The feedback mechanism of Model risk should incorporate the ‘Risk Appetite’ and ‘Tolerance’ levels as prescribed in the Risk Management policies

Model risk needs to be quantified for the enterprise which means it needs to be aggregated for all models across different types of risks faced by the bank