Introduction:
Financial models, and misuse of them, have been blamed for contributing to recent economic woes. Public perception is that default of incorrectly modeled, complex financial securities played a major role in failures of banks, insurers[1], investment banks[2], and GSEs[3], and that foreclosures of poorly underwritten (using faulty credit models[4]) loans backing the securities precipitated large housing price declines. Many of the securities that defaulted carried the highest level of bond ratings when issued. Widespread subsequent bond defaults suggest that rating agency models were either flawed, used inappropriately, or both.
Models are intended to be simplified representations of real-world relationships among observed characteristics, values and events. Ironically, this simplification involves using complex quantitative methods, systems, or approaches, that:
- Apply statistical, economic, financial, or mathematical theories, techniques, assumptions, and expert judgments;
- Process input data; and
- Produce quantitative estimates.
Understanding model strengths and limitations, and how to interpret and use outputs requires an understanding of the methods and assumptions used, as well as sources and content of input data processed.
Firms and regulators have recognized that use of complex models is necessary and beneficial, but the modeling process has many risks that must be managed. This article explores modeling risks and the best practices firms use to manage these risks. First, the components of the modeling process are identified and discussed. Then tables showing sources of risk in the modeling process and means to control those risks are presented.
Models and the Modeling Process
Common model use categories include estimating value, projecting risk exposure, analyzing business strategies, determining regulatory capital needs and allocating internal costs. Table 1 itemizes typical applications within each of these categories.
[1] Bond insurers, primary mortgage insurers, general insurers and credit default swap writers are included in this category.
[2] Examples include Lehman Brothers and Bear Stearns.
[3] GSE is the abbreviation used for government sponsored entities, such as Fannie Mae and Freddie Mac.
[4] See Table 2 below
Models in each category carry risk. Model risk is defined many ways, a selection of which is in the Appendix. For discussion here, model risk is defined as “the potential for adverse consequences from decisions made or actions taken based on outputs or reports created by a model that was incorrectly developed, implemented, populated[5], used, and/or validated.”
The adverse consequences can be financial, regulatory, reputational, competitive, and legal. Table 2 is a summary of some recent consequences from failures in model risk management.
[5] Populated refers to loading assumptions and data into the model.
These recent consequences have affected market and regulator views towards model risk management. For instance, until 2011, banking regulators focused attention on model validation.[6] Their goal was to make sure models were making reasonable predictions compared with actual experience.[7]
With the Federal Reserve’s and OCC’s promulgation of SR 11-7 and its related Attachment, the regulators broadened their focus and guidance. SR Letter 11-7 Attachment identifies the following as key to effective model risk management:
- Sound development, implementation, use and validation of models (“modeling activities”);
- Governance and control mechanisms over modeling activities.
Taking the regulatory approach, the following diagram shows the major components of the modeling process and how they are linked. Governance and control mechanisms touch all components and are core to the overall process.
The modeling process starts with Development. Development consists of the following key steps:
- Identify for what the model will be used;
- Create model specifications;
- Construct the model;
- Test the inner model workings: logic and mathematics, user interfaces, data transfers; program calls, speed, memory usage, etc.;
- Create extensive documentation of the model, its purpose and intended usage.
[6] For instance, see SR 09-1, OCC 2000-16, 12 CFR 3, Appendix C, 12 CFR 208 Appendix F, 12 CFR 225, Appendix G, the Federal Reserve’s Trading and Capital-Markets Activities Manual and certain subject matter booklets of the Comptroller’s Handbook.
[7] Some authors prefer to use the term validation for assessing the reasonableness of assumptions used in the model, and “verification” for making sure the model makes reasonable predictions. See for example, Morini, M. Understanding and Managing Model Risk: A Practical Guide for Quants, Traders and Validators.
Once the model is created, it must be implemented. Implementation steps include:
- Test the model using actual test data and assumptions;
- Perform extensive review of outputs and reports produced from test data and assumptions;
- Update and calibrate the model based on test results;
- Develop training materials and conduct training.
Development and implementation have limited initial timelines, often set by transaction needs and budget considerations. However, some of the development and implementation steps are periodically repeated as model updates are conceptualized, programmed and implemented.
The next two steps in the modeling process, usage and validation, are ongoing. Model Usage consists of the following primary steps.
- Enter assumptions and run the model using live data;
- Obtain, interpret and question results;
- Modify assumptions and perform reruns as needed or desired;
- Make decisions and take actions based on results.
These steps are repeated each time the model is run.
As use of the model becomes ongoing, so does Validation by both users of the model and independent parties. The regulators combine into validation two traditional concepts:
- Verification – verifying that model assumptions are turned correctly into numbers;
- Validation – assessing if the assumptions of the model are valid.
The parties involved and key steps are as follows:
- Independent parties (with no ownership stake in the model) perform independent reviews of the model’s logical and conceptual soundness, examine assumption development and usage, and evaluate model outputs and their usage compared to actual results and intended usage;
- Parties running and using the model continually test projected outputs versus actual experience and recalibrate the model and modeling assumptions;
- Both groups of validating parties provide feedback to designated control personnel.
Validation includes back-testing of prior projected amounts against actual performance as well as comparing model outputs to proxies. Experienced analysts, operations personnel, traders and others also critically evaluate the reasonableness of results. The central core elements of the modeling process, described generally as Governance and Control Mechanisms, provide the framework and controls for the other processes. Governance starts with the board of directors and senior management and is spread throughout the organization through formal policies and procedures. Control mechanisms involve defining ownership, control and compliance roles within the organization, enforcing responsibilities for parties within these roles, maintaining structure for and performing independent reviews, and maintaining control over use of external resources.
The next section will examine risks and controls on each component of the modeling process.
Sources of Risk throughout the Modeling Process and Means of Controlling the Risks
The tables in this section focus on each of the major components of the modeling process, some common sources of risk encountered in the modeling process and means of controlling those risks. They are meant to be illustrative, not exhaustive.
Data Input
[8] The “Validation” table draws significantly from OCC 2000-16.
[9] The “Governance” table, and each subsequent table presented, draws significantly from SR Letter 11-7 Attachment.
Conclusion
Model risk is the potential for adverse consequences from decisions made or actions taken based on outputs or reports created by a model that was incorrectly developed, implemented, populated, used, and/or validated. Companies cannot eliminate model risk, but they can manage it and attempt to minimize negative consequences from use of models in running their businesses. Model risk management starts with the board of directors and senior management, but requires a firm-wide focus and commitment to mitigate risk throughout the entire modeling process.
Best practices in model risk management continue to evolve as businesses become more complex and the models used, which are designed to be simplified representations of the real-world relationships encountered, likewise become more complex. It is important to manage risk throughout the entire modeling process rather than just focus on validating output of the model itself. The modeling process includes sound development, implementation, use and validation of models, with governance and control mechanisms over all activities.
The tables included herein are a starting point from which organizations can assess their model risk management capabilities, and from which to enhance and build them. The success of each entity in managing the modeling process will be manifested through greater profitability, stability and efficiency, and less financial, regulatory, reputational, competitive, and legal challenges.
