white paper model risk sept 2011

White Paper: Model Risk

Version: Final September 22, 2011

2 Introduction

Table of Contents New Supervisory Guidance on Model Risk Management ........................................................... 2

Introduction ............................................................................................................................ 3

Definitions .............................................................................................................................. 3

Model Usage .......................................................................................................................... 4

Better Practices Guidance ...................................................................................................... 5

Model development, implementation, and use ........................................................................ 5

Development ....................................................................................................................... 5

Implementation ................................................................................................................... 6

Use ..................................................................................................................................... 6

Model verification and validation ............................................................................................. 6

Model Risk Governance ......................................................................................................... 7

A Note on Third-Party Models ................................................................................................ 7

Self-Assessment of Your Model Risk Management Practices ................................................. 8

Conclusion ............................................................................................................................. 9

Authors ................................................................................................................................... 9


3

Introduction In early April 2011, the Office of the Comptroller of the Currency (OCC) and the Board of Governors of the Federal Reserve System (FED) released Supervisory Guidance on Model Risk Management (the Guidance or OCC 2011-12). From our perspective, there was little new in the guidance. However, OCC 2011-12 codifies better practices that were previously familiar only to sector specialists, including the Capital Markets teams at the OCC and FED. So the news is not that there is anything new, per se, rather the news is that better practices have been clearly communicated to OCC- and FED-supervised banks and thrifts via OCC 2011-12i. The Guidance reinforces prior regulatory recommendations. From a regulatory perspective, this guidance was issued to “provide comprehensive guidance for banks on effective model risk management”ii. An unwritten cause of this issuance was that bank supervisors have observed substandard model risk practices across the industry at institutions both large and small. The Guidance notes that the use of models, especially complex models, varies across the industry, so model risk management and mitigation should be commensurate with risk and model complexity. As is typical of regulatory issuances, it notes that internal controls, policies, and documented procedures are appropriate. The FDIC and NCUA were not included in this issuance, as they were with the Interagency Advisory On Interest Rate Risk Management (OCC 2010-1a) and the Interagency Policy Statement on Funding and Liquidity Risk Management (OCC 2010-27a). As with OCC 2000-16 (Model Validation) and SR 09-1 (Application of the Market Risk Rule in Bank Holding Companies and State Member Banks), it is likely that FDIC and NCUA supervisors will apply OCC 2011-12 to supervised institutions.

Definitions A model is defined as “a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates”. The definition of model is expanded to include “quantitative approaches whose inputs are partially or wholly quantitative or based on expert judgment, provided that the output is quantitative in nature”.

The Guidance, congruent with OCC 2000-16, delineates three components of a model:

1. An information input component, which delivers data and assumptions to the model

2. A processing component, which transforms input into estimates

3. A reporting component, which transforms estimates into useful business information. The Guidance suggests there are two main sources of model risk:

4 Model Usage

1. Use of a model with errors, including: a. an incorrect theoretical foundation b. improper implementation or maintenance c. inaccuracies in any of its three components (as above).

2. Incorrect or inappropriate use of a model a. Use a model in inappropriate circumstances b. Results that are not decision makers c. Lack of explanation or understanding of limitations

The Guidance defines model risk as “the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports”. This is a good summary definition as it also includes the potential for poor decisions based on the outputs of technically correct, fully validated models that were not totally understood by decision makers (e.g. Value at Risk models, 2. above).

Eugene Derman’s 1996 articleiii on Model Risk provides more information on the types of model risk:

• Wrong model o Inapplicability of model o Incorrect model specification

• Model implementation o Programming errors o Technical errors o Use of numerical approximations

• Model usage o Implementation Risk o Data issues o Calibration errors

Two of the primary tools for effectively managing model risk are model verification and model validationiv. Model verification refers to a process aimed at answering the question “did we build the model right”. Model validation refers to a process aimed at answering the question the question “are the results credible?

Model Usage Industry observers and the Guidance recognize that the use of models in banks is pervasive in pricing, risk management, and reporting:

• evaluate and underwrite credits

• price loans and deposits

• create and analyze forecasts and strategies


5

• measure and monitor credit, liquidity, market, and operational risks

• value exposures and financial instruments

• report and disclose results

The Guidance states, “The use of models invariably presents model risk”. Derman noted, “This reliance on models to handle risk carries its own risks”. The authoritative Verification, Validation, and Accreditation of Army Models and Simulationsv simply say, “There is no such thing as an absolutely valid model”. Accordingly, we would suggest that best practices model risk management practices recognize that model risk is unavoidable, but that effective model risk management mitigates model risk within a cost/benefit context.

Better Practices Guidance The Guidance highlights three areas for effective, best practices risk model management:

• Model development, implementation, and use • Model validation • Governance, policies, and control

While the above framework is not new to bankers, the Guidance notes “A guiding principle for managing model risk is "effective challenge" of models, that is, critical analysis by objective, informed parties who can identify model limitations and assumptions and produce appropriate changes.”

The notion of “effective challenge” is perhaps new to all but the largest of banks. In very large banks, a specialized Model Risk or Model Validation unit reports to the Chief Risk Officer. In most banks, Internal Audit staff or the Chief Financial Officer’s team is tasked with model review. Usually, these units lack the expertise necessary to review the theoretical underpinnings and math of complex models. As a result, the review is ineffective, and frequently relies solely on checklists emphasizing balancing to financials and policy compliance, which while necessary are not sufficient.

Model development, implementation, and use This topic is broadly consistent with the definition of model verification, as defined above. Examples from the ALM model sector may illuminate the notion of “effective challenge” in this area.

Development As regards vendor model development and documentation, we have found that some models, even those with hundreds of client banks and that have been “certified” by a third party, have

6 Model verification and validation

fundamental valuation flaws. The flaws have been noted in both the valuation model math and in the documentation of how to implement the valuation tools. Perhaps this is due to the greater subject expertise of the model verification and validation team than the model development or certification teams. In any event, it is unlikely that community bank Internal Audit teams have sufficient expertise in valuation models to “effectively challenge” the use of a vendor valuation model or module, so the use of outside experts is important.

Implementation We have also observed inadequate implementations of otherwise sound risk models. Recently, at a $1+ billion National Bank, we asked the ALM staff about the model implementation and subsequent model calibration. The staff readily noted that they had not updated its pricing, valuation, and prepayment configuration, including assumptions, over the past ten years since initial vendor implementation “because the model did it automatically”. Unfortunately, or fortunately, depending on your perspective, for the bank and readers of this publication, the state of the art over the past decade is such that no ALM model automatically updates every assumption appropriately and automatically.

Use One of the key assumptions in the use of an ALM model is Core Deposit rate sensitivities. In many banks, unsubstantiated and undocumented rate sensitivity estimates are used. This has the impact of suggesting that their bank is either “asset and/or liability sensitive”. The use of alternative assumptions, via sensitivity analysis, market or institution statistical studies, or other quantitative approaches may affect the magnitude and/or directionality of bank risk exposures. Again, the typical Internal Auditor is unfamiliar with the risk reporting consequences of key assumptions, let alone the impact such changes may have on product pricing, hedging strategies, and balance sheet allocation decisions.

Model verification and validation The Guidance outlines three core elements of an effective validation:

1. Evaluation of conceptual soundness and assessment of the quality of the model design and construction

2. Ongoing monitoring, including process verification and benchmarking 3. Outcomes analysis, including back-testing


7

The topic of Model Validation has been covered in previous BALM articles, including ours, and will likely be the topic of futures articles, including ours. However, we would note that models are used in other parts of the world in addition to banking and present an example vi of verification and validation activities. Accordingly, we suggest that “model validation” as used in Banking typically refers to “model verification and validation” in other sectors.

Model Risk Governance To control model risk, banks need a strong, thorough model governance process. This includes outlining the important roles for the Board, senior management, the model operators/owners and the model validators. The key is defining a workable structure that ensures the integration of high quality model forecasts with management decision making and regulatory requirements. The challenge is to do this in a way that does not create a system crippled by micromanagement processes. The following summarizes key requirements for effective governance:

• Board of Directors and Senior Management - establish a strong model risk management framework that fits into the broader risk management of the organization

• Policies and Procedures – require model risk management activities be formalized through policies and documented procedures to implement them

• Roles and Responsibilities – address ownership, controls and compliance • Internal Audit - assess the overall effectiveness of the model risk management

framework • External resources – specify in writing third-party resources engaged to assist with

model validation and review, compliance functions, or other activities in support of internal audit

• Model inventory – require the maintenance of a comprehensive set of information for models implemented for use, under development for implementation, or recently retired where applicable

• Documentation – maintain detailed documentation of model development and validation so that parties unfamiliar with a model can understand how the model operates, its limitations, and its key assumptions.

A Note on Third-Party Models Some banks rely on vendors to run financial models and prepare risk reports for them. The guidance addresses the validation of vendor and third-party products by stressing that although the process may be somewhat modified, the validation principles should be the same as applied to in-house models. This helps bank management better understand the vendor product and its capabilities, applicability, and limitations.

8 Self-Assessment of Your Model Risk Management Practices

The Board and bank management should ensure that the model works effectively if they use reports to make decisions. Management should be diligent in tracking changes the vendor makes to the model including for each reporting period a list of any model changes and an explanation of the expected effects on the resulting reports.

Self-Assessment of Your Model Risk Management Practices Developing, implementing, and maintaining an effective model requires ongoing diligence in oversight, validation and controls management. Ask yourself, or have your Internal Audit ask these questions for each model your bank uses:

1. Are the model results significant to managing your bank’s financial position and/or defining future strategy?

2. Does the model have a sound theoretical foundation? 3. Are the assumptions reasonable and supported? 4. Do senior management and the Board understand the model’s primary purpose,

methods, and calculations? 5. Are model limitation regularly communicated and understood? 6. Are the data inputs accurate? 7. Are model results translated into understandable analytical reports that support decision

making? 8. Has your model been validated within the last twelve month period? 9. If your model has been validated, have significant findings and recommendations been

addressed? 10. If the model is a third-party model, then answer these additional questions:

a. Do you have an appropriate process in place for selecting vendor models? b. Do you require that the vendor provide developmental evidence explaining the

product components, design, and intended use, to determine whether the model is appropriate for your bank’s products, exposures, and risks?

c. Does your vendor regularly provide appropriate testing results, showing their product works as expected?

d. Is your bank clearly aware of the model’s limitations, detailed assumptions and where the model’s use may be problematic with respect to your particular products?

e. Does your vendor regularly report on and conduct ongoing performance monitoring and outcomes analysis, and make appropriate modifications and updates over time?

f. Are your bank’s model customization choices documented, justified and understood?

g. If your model vendor provides input data or assumptions, are you provided with details and is the relevance to your bank’s situation understood and assessed?

h. Are you conducting ongoing monitoring and outcomes analysis of vendor model performance using the bank’s own outcomes?


9

i. Do you have an in-house model expert who understands the systematic

modeling procedures that can help the bank understand the vendor product and its capabilities, applicability, and limitations?

Conclusion The recent Guidance alerts banks to the emphasis that the OCC and FED are placing on effective model risk management. Over the past two years, we have seen an increase in formal and informal regulatory orders related to effective model risk management for market, credit, and liquidity risk models. These orders have also noted the effectiveness and frequency of Board, ALCO, and Credit Committee reporting. We suggest that you use the “Self-Assessment” above and review the Guidance as time allows.

Authors Fred Poorman Jr., CFA, Managing Principal, Bank Risk Advisors Howard Stern, PhD, Senior Consultant, Bank Risk Advisors Terry Treadwell, CPA, Senior Consultant, Bank Risk Advisors i Existing regulatory guidance includes– the FDIC’s Supervisory Insights (Winter 2005) article on Model

Governance; OCC Bulletin 2000-16, the current guiding directive on ALM model verification for national

banks, as well as mandates outlined in various other regulatory directives -- the most recent IRR advisory,

2010-1A, and discussion in FDIC's Supervisory Insights (Winter 2009).

ii Unless otherwise noted quotations are from the Guidance

iii http://www.ederman.com/new/docs/gs-model risk.pdf

iv http://ppc.documents/model-verif-validation-full.pdf

v http://www.apd.army.mil/pdffiles/p5 11.pdf

vi http://ppc.documents/model-verif-validation-full.pdf

white paper model risk sept 2011

Business