Model risk management meets LLMs#
US banking supervisors set out model risk management expectations in SR 11-7 in 2011 [1]: a model inventory, conceptual soundness review, independent validation with "effective challenge", outcomes analysis and ongoing monitoring. In April 2026 the agencies replaced it with revised interagency guidance, SR 26-2, which keeps those core principles and places more emphasis on proportionality and model materiality [2]. Whichever version your examiners use, LLM applications strain three of its assumptions:
| MRM assumption | Why LLM systems break it | What to do instead |
|---|---|---|
| Outputs are numbers you can backtest | Outputs are text and actions; "correct" is a rubric judgement | Validated graders with measured agreement; see LLM-as-a-judge |
| The model changes only when the bank changes it | Vendor model updates change behaviour silently | Pin versions; rerun the full suite on any vendor change; treat it as a model change |
| Conceptual soundness can be reviewed from documentation | Foundation-model internals aren't available | Shift weight to outcomes testing, adversarial testing and runtime monitoring |
Use cases and their dominant risks#
| Use case | Dominant risk | Primary evaluation | Primary control |
|---|---|---|---|
| Customer-service agent with account actions | Unauthorised transactions via injection or social engineering | Injection corpora per channel; action assertions | Tool-call policy, step-up authentication, limits |
| Credit decision support | Discriminatory or unexplainable outcomes; regulatory classification | Disparity metrics across protected groups; reason-code faithfulness | Human decision-maker; documented reason codes |
| Fraud and AML analyst copilots | Hallucinated facts in case narratives; data leakage across cases | Groundedness graders; cross-case retrieval probes | Case-scoped retrieval; citation requirements |
| Advice and wealth assistants | Unsuitable recommendations; over-refusal of legitimate questions | Expert-written suitability rubrics; refusal calibration | Scope restrictions; adviser review for personalised recommendations |
| Internal code and ops agents | Secrets exposure; destructive actions in production | Agent red teaming in sandboxed environments | Read-only defaults; approvals; secret scanning |
In the EU, AI systems used to evaluate the creditworthiness of natural persons or establish their credit score are listed as high-risk in Annex III of the AI Act [3], with obligations now applying from 2 December 2027; see our timeline. Life and health insurance risk assessment and pricing for natural persons are also listed.
Payment-capable agents: a worked control design#
For an agent that can initiate payments or change payees, we design so that no single model decision can move money to a new destination:
- Separate tools for "pay existing payee" and "add payee", with the latter requiring step-up authentication outside the conversation.
- Provenance rule: payment tools can't be triggered by instructions whose provenance is a document, email or web page.
- Velocity and amount limits enforced by the payment system, not by the prompt.
- Out-of-band confirmation that shows the customer the raw request details, not the agent's summary.
The quantitative reason: at realistic per-attempt injection success rates, attackers who can send unlimited messages reach high cumulative success (see retry economics). Only controls that don't depend on the model's judgement bring the residual risk to a level a risk committee can sign.
What an LLM validation report should contain#
- Use-case scope, materiality tier and the unacceptable outcomes it was tested against.
- Eval categories with metrics, intervals and sample-size justification; see sample sizes.
- Grader validation results against qualified reviewers (compliance officers, credit analysts).
- Adversarial testing results mapped to the OWASP LLM Top 10 [4], with retest evidence.
- Vendor-change protocol and monitoring thresholds.
- Limitations and compensating controls, stated plainly; effective challenge depends on them.
Frequently asked questions#
Does SR 11-7 apply to large language models?
Supervisors have treated LLM-based tools used in decision-making as models within model risk management. SR 11-7 was replaced in April 2026 by revised interagency guidance (SR 26-2), which keeps the core principles of validation, effective challenge and monitoring while emphasising proportionality to model materiality.
Is credit scoring high-risk under the EU AI Act?
Yes. AI systems intended to evaluate the creditworthiness of natural persons or establish their credit score are listed in Annex III as high-risk, with exceptions such as fraud detection. Following the 2026 Digital Omnibus, Annex III obligations apply from 2 December 2027.
Sources#
- Board of Governors of the Federal Reserve System and OCC. SR 11-7: Supervisory Guidance on Model Risk Management, 4 April 2011. federalreserve.gov
- Baker Tilly. Updated interagency guidance on model risk management (summary of SR 26-2, April 2026). bakertilly.com
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Annex III. eur-lex.europa.eu
- OWASP GenAI Security Project. OWASP Top 10 for LLM Applications 2025. genai.owasp.org