Building interpretable AI models: a model risk management challenge - by Ushasi Sengupta & Sanjukta Dhar

building interpretable AI models: a model risk management challenge

by Ushasi Sengupta & Sanjukta Dhar

introduction

Steered by demand for automation, efficiency, and personalization and availability of intense and diversified data, financial institutions have started leveraging AI across the value chain. An IDC spending guide says that global spending on artificial intelligence (AI) is going to double over the next four years, reaching more than $110 billion in 2024.1 Not just in financial decision making, AI models are present across different touchpoints - customer acquisition, detection of fraudulent transactions, emotional analytics, and leveraging alternate (Alt-data) datasets associated with financial transactions. As organizations are adopting AI rapidly in various ways, the demand for nimble decisions and accuracy in machine driven complex decision-making is also growing. So does usage of black box AI systems.

This, in turn, necessitates the need for fair decision engine that will help organizations protect their reputation and customer base from potential vulnerabilities of opaque decision models. These models, due to lack of interpretability, may have unprecedented financial implication as well as impact on goodwill. To avoid this, financial and risk models need to be reoriented such that they be more: 1) Transparent 2) Auditable 3) Analogous and 4) Explainable.

To attain these attributes, institutions need to reimagine the traditional Model Risk management framework.2 The governance plan to ensure fair decision systems needs to span across model data & algorithm, model responses and business impacts. These new governance aspects of model risk management components will include: 1) Model oversight and Control 2) Model Outcome Validation 3) Model Data Validation.

This article will explore various methodologies to reorient model risk management towards this desired state so that complex financial models can be made more interpretable.

1 / https://www.idc.com/getdoc.jsp?containerId=prUS46794720

2 / https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf

black-box to white-box – looking at the checkboxes / critical attributes

re-imagining the model risk management components – our propositions

As mentioned above, an opaque decision engine needs to possess 1) Transparent 2) Auditable 3) Analogous and 4) Explainable traits in order to be transformed into a white-box model.

1. Transparent: A transparent AI model should be able to answer the audience the system functions i.e. how and why a system behaves in certain manner. Stakeholders of any AI model – developer and users – need to know about the model. Creating transparent AI models helps build an AI system with a clear understanding of its behavior and attributes.

2. Auditable – AI auditability ensures that an independent authorized user can validate model behavior across different use cases. This will help in periodic assessment of the model risk and reliability, by identifying consistency of decisions, responses in outlier scenarios, impact of marginal changes in broader ecosystem, and possible recovery recommendations.

3. Analogous – Outcome of a black-box model needs to be benchmarked with that of a similar white-box model. In this case, the white-box model could be a reference model with analogous outcome to validate the correctness.

4. Explainable – AI explainability describes different factors that contribute to AI decision systems.

It can be broadly categorized into two broad sections – Pre-modeling explainability and Postmodeling explainability.

a. Pre-Modeling Explainability – In a pre-modeling stage, data is the most important aspect.

So, understanding data – meta data, data structure - data classification, summarization, clustering, and relationship play significant roles. Different methods such as EDA, K-means

Clustering, and feature engineering help understand the dataset.

b. Post-modeling Explainability – Post-model explainability is backward tracing of model outputs. One of the methods of post-modeling explainability is Shapley Score. It determines the marginal score of each parameter depending on their contribution. Sensitivity Analysis is another method that identifies parameter sensitivity by varying values and observing model outcome deviation.

As discussed above, institutions need to implement model risk management in a complex model ecosystem to ensure fair decision engine and a white-box AI system. The governance aspects are inter-dependent and span across a micro-level structure to the organization level macro oversight. As mentioned above, three pillars of the model risk management are Data Validation, Outcome Validation and Model oversight and Control.

Figure 1- Model interpretability value chain and its key components

1. Data Validation – This is part of the 1st line of defense of model risk management. Knowing data becomes very important as a model can carry forward the biasness of data set in its decisions. Specific business challenges can also arise due to an imbalanced data set. So, Data Validation needs to take care of data discovery, preprocessing and feature engineering.

Let’s take an example of a Financial Lending model that classifies the credit worthiness of a customer, by analyzing a customer’s demographic (age, gender), financial (credit history, annual income), behavioral (criminal record, employment history), and biometric information. Data discovery or feature engineering intends to pick up the most critical parameter from the input parameters. So, in our case it can be credit history, age, and annual income. By analyzing data and understanding its pattern, a model can also decide on the algorithm to select. So, for identifying customers’ credit worthiness, the best out of Decision Tree/ Random forest/K-means clustering could be used as it is a classification (creditworthy or NOT creditworthy) problem. Thus, entire pre-modeling explainability comes under Data Validation purview.

2. Outcome Validation – Model outcome validation is part of both 1st and 2nd lines of defense of model risk management. Components of post-modelling explainability are part of model outcome validation. A model might face periodic decay due to data drift or concept drift. Governance aspects around Model provenance can be established to document the dependencies of various data sets, data lineage and their impact on model materiality. Also, sensitivity analysis, SHAP score can help model certify fit or re-calibrate if necessary.

3. Model oversight and control - Considering this wide spread of complex AI models, organization needs to define a set of model design criteria and periodic review and rating system through model oversight and control. The various tenets of overall model oversight are Transparency, Auditability, Explainability, Analogy, Accountability and Fairness.

The model governance framework needs to be conceptualized and operationalized as per the interest of stakeholders and business.

This article portrays an overarching framework of managing and remediating potential vulnerabilities of AI based financial black-box models. This opens avenues to probe further the internal layers of governance. In our subsequent discussions we will explore each of these individual components and governance aspects of their respective modules.

authors

Ushasi Sengupta

Tata Consultancy Services Ushasi Sengupta is a senior Research Analyst in corporate functions of Tata Consultancy Services India. Her responsibilities revolve around research insights and advisory, catering through the customer journey in the Banking and Financial Services value chain.

She has been engaged in exploring new business opportunities and technology and industry trajectory. She is also enthusiastic about recent developments and technology risks in financial domain and has shared her thoughts through different publications.

Sanjukta Dhar leads the Risk and Regulatory Compliance practice of Tata Consultancy Services for Canada Geography.

Sanjukta comes with 18 years of domain and technology experience across Market and Counterparty credit risk modelling, aggregation and reporting functions for major banking institutions. She has led and participated in many critical build-the-bank risk & regulatory compliance programs such as Risk finance Integration, FRTB Standardized approach, VaR Back Testing framework, BCBS239 and SR11/7.

She is based out of Toronto and frequently writes/talks about applied analytical tools and techniques (Data Science, Machine learning, MLOps) in the Financial Risk management domain.

Sanjukta Dhar

Tata Consultancy Services