Three frameworks, three kinds of obligation#
| EU AI Act | NIST AI RMF + AI 600-1 | ISO/IEC 42001 | |
|---|---|---|---|
| Type | Regulation, Regulation (EU) 2024/1689 [1] | Voluntary framework (US) [5][6] | International management-system standard, certifiable [7] |
| Who it binds | Providers and deployers placing AI on the EU market, by risk class | Anyone who adopts it; often referenced in contracts and US policy | Organisations seeking certification |
| Testing language | Accuracy, robustness and cybersecurity (Art. 15); risk management with testing (Art. 9); adversarial testing for systemic-risk GPAI (Art. 55) | The Measure function: evaluate trustworthiness characteristics; AI 600-1 adds GAI-specific actions including red teaming | AI risk assessment, controls and performance evaluation within a plan–do–check–act management system |
| Evidence style | Technical documentation, declared accuracy metrics, conformity assessment | Documented measurements, tracked over time | Audited records of process and results |
One test programme, three evidence trails#
The practical insight is that a well-designed evaluation programme produces the evidence for all three at once. The mapping we use:
| Activity | EU AI Act | NIST AI RMF / 600-1 | ISO/IEC 42001 |
|---|---|---|---|
| Threat model and unacceptable outcomes | Art. 9 risk identification | Map function | AI risk assessment |
| Release eval suite with declared metrics and intervals | Art. 15(1)–(3) accuracy and declared metrics | Measure: validity and reliability | Performance evaluation |
| Adversarial red teaming (injection, misuse) | Art. 15(5) cybersecurity; Art. 55 adversarial testing (GPAI systemic risk) | Measure; AI 600-1 red-teaming actions | Risk treatment verification |
| Runtime guardrails and logs | Art. 12 record-keeping; Art. 14 human oversight | Manage function | Operational controls |
| Online evaluation and incident tracking | Post-market monitoring; serious-incident reporting | Manage: monitoring | Monitoring, nonconformity and corrective action |
| Grader validation, data provenance | Art. 10 data governance (high-risk) | Measure: measurement validity | Data for AI systems controls |
What auditors actually want to see#
- A metric definition that a third party can recompute: dataset version, grader version, sampling, and the statistical method behind the interval.
- Trend over versions, not a single snapshot, showing that regressions are detected and acted on.
- Adversarial coverage mapped to a recognised taxonomy (OWASP LLM Top 10, MITRE ATLAS) with residual-risk statements.
- Control effectiveness: guardrail block and false-positive rates in production, not just their existence.
- Change records: who approved each release and on what evidence, including gate overrides.
Each of these is a by-product of running evals and red teaming as engineering processes. Producing them after the fact for an audit costs far more than generating them continuously.
Provider or deployer: which obligations are yours#
| Your situation | Likely AI Act role | What that means for testing |
|---|---|---|
| You train or substantially modify a foundation model and offer it to others | GPAI model provider (systemic-risk duties if designated) | Model evaluations incl. documented adversarial testing; incident reporting |
| You build a credit-scoring or hiring tool on a third-party model and sell it | Provider of a high-risk AI system | Full Article 9/15 testing, technical documentation, conformity assessment |
| You use such a tool internally in the EU | Deployer | Use per instructions, human oversight, monitoring, log retention; evidence that the provider's declared metrics hold in your context |
| You build a customer-support agent (not in Annex III) | Provider of a non-high-risk system | Transparency duties where applicable; testing is driven by your own risk and contracts, and NIST/ISO evidence is usually what customers ask for |
Role is decided per system, and one company can hold several roles. "Substantial modification" of a high-risk system can turn a deployer into a provider, which is one reason to keep fine-tuning and prompt changes under the same change control as code.
Worked example: one release, three evidence trails#
A lender ships version 3.2 of an LLM-assisted credit-decision support tool used by underwriters in the EU and the US. One release pipeline produces:
- Eval report: 14 categories with Wilson intervals, paired tests against v3.1, disparity metrics across protected groups with intervals, and grader validation against underwriter labels. This is Article 15 declared-metrics evidence, NIST Measure evidence, and ISO 42001 performance-evaluation records.
- Red-team report: injection through uploaded bank statements, extraction of other applicants' data, reason-code manipulation, mapped to OWASP IDs and retested. This covers Article 15 cybersecurity, NIST AI 600-1 red-teaming actions and ISO risk-treatment verification.
- Change record: approver, gate results and one documented override with rationale. This is Article 9 lifecycle evidence, NIST Govern accountability and ISO operational control.
- Monitoring plan: online evaluation sampling, drift thresholds, incident path. This covers post-market monitoring, NIST Manage and ISO monitoring and improvement.
None of these documents was written for an auditor. They are what a careful engineering team produces anyway, which is why we treat compliance as a by-product of measurement rather than a separate workstream.
Common misconceptions#
- "Our model vendor is compliant, so we are." Vendor evaluations cover the model, not your prompts, tools, data and users. Your system's behaviour is yours to evidence.
- "The high-risk deadline moved, so we can wait." Conformity evidence needs a history of testing across versions. Starting a year before the deadline leaves one or two release cycles of evidence.
- "A red-team report is enough." Every framework asks for ongoing measurement and monitoring, not a point-in-time assessment.
- "ISO 42001 certification equals AI Act compliance." It demonstrates a management system. It doesn't replace system-specific technical requirements.
Where to start#
If you are early: write the evaluation plan first, because it forces the threat model, metrics and owners that every framework asks for. Then run a red-team assessment mapped to the OWASP Top 10, and stand up runtime logging. The two deep dives cover the specifics: EU AI Act testing requirements and timeline and NIST AI RMF Measure for generative AI.
Frequently asked questions#
Does the EU AI Act require red teaming?
For providers of general-purpose AI models with systemic risk, Article 55 requires model evaluation including conducting and documenting adversarial testing. For high-risk AI systems, Article 15 requires appropriate accuracy, robustness and cybersecurity, including resilience against attempts to exploit vulnerabilities, which in practice calls for adversarial testing.
Is ISO/IEC 42001 the same as the EU AI Act?
No. ISO/IEC 42001 is a voluntary, certifiable management-system standard published in December 2023; the EU AI Act is binding regulation. Certification can support, but does not by itself demonstrate, AI Act compliance.
Sources#
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Official Journal of the European Union, 12 July 2024. eur-lex.europa.eu
- EU AI Act, Article 15: Accuracy, robustness and cybersecurity (AI Act Service Desk, European Commission). ai-act-service-desk.ec.europa.eu
- EU AI Act, Article 55: Obligations for providers of general-purpose AI models with systemic risk. artificialintelligenceact.eu
- Digital Omnibus on AI: amendments to the AI Act timeline (overview and text tracker). artificialintelligenceact.eu
- NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 2023. doi.org
- NIST. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, July 2024. doi.org
- ISO/IEC 42001:2023, Information technology: Artificial intelligence: Management system, published December 2023. iso.org