Last updated:
AAIA preparation guide
AAIA evidence trap: a document is not proof the AI control works
Direct answer
The trap is accepting a design document as proof that an AI control works. A model card, an AI policy or a validation plan shows what was intended. Evidence of operation shows what happened: monitoring alerts and their follow-up, review and override logs, change tickets linked to model versions, and your own re-performance. CertArc's frame: name the control, the period and the record it should leave, then test that record.
ISACA's outline gives evidence its own subarea, 3C Audit Evidence Collection Techniques, beside 3D Audit Data Quality and Data Analytics. One of ISACA's tasks is to “evaluate the design and effectiveness of controls specific to AI.”
Design documents still matter: they tell you what the control should do and what to test. They are where the audit starts, not where it ends.
Start AAIA baselineDesign evidence versus operating evidence
| Tempting evidence | Why it feels right | What shows the control operated |
|---|---|---|
| A model card | It is the model's official documentation | Validation results for the deployed version, and monitoring against the limits the card states |
| An AI policy that requires human review | Senior management approved it | Review and override records for a sample of decisions from the period |
| A supplier's assurance report | An independent firm wrote it | Its scope, period and exceptions checked against your use, and tests of the controls you run |
| A screenshot of a monitoring dashboard | It shows the metric is tracked | The alert history for the period and the tickets showing someone acted |
| Management's statement that retraining is controlled | Management knows its own process | Model registry versions reconciled to approved change tickets |
Original teaching example
A lender's AI policy says a person reviews every application the credit model declines above a set amount. The model card describes the review step, and the risk team gives you a process flowchart. You are asked whether the human-review control is operating.
All three are design evidence. To conclude on operation, take the declined applications above the amount from the system's own records for the audit period, check the population is complete, then sample. For each item, look for a review record: who reviewed it, when, and whether they could and did overturn the model. If no records exist, the control cannot be shown to operate, and that is the finding.
The trap
A well-written document answers the design question, not the operation question.
Options that rely on a model card, a policy or a signed-off plan feel safe because the documents are official and complete. But the question asks whether the control worked, and a document cannot show that.
Why the tempting answer fails
Documents describe intent, and AI systems change after the documents are written.
A model is retrained, its input data drifts, a threshold is changed or a supplier updates its model. A model card written for the first version may describe a system that no longer runs. Check which version the document covers before relying on it at all.
What sufficient evidence looks like
Evidence that is reliable and relevant to this model version and this period.
- System-generated records rather than prepared summaries.
- Coverage of the whole audit period, not one good day.
- A link to the model version in scope.
- A complete population before sampling.
- Your own re-performance where you can, such as re-running a sample of inputs.
AI-processed evidence needs checking too
If an AI tool produced or filtered the evidence, test its accuracy before relying on it.
That includes an AI tool the audit team uses itself. ISACA's tasks include evaluating the impacts, opportunities and risk of integrating AI solutions within the audit process.
Where this appears in the outline
Mainly 3C and 3D, applied to the controls in Domain 2.
In CertArc's reading, the trap is common where Domain 2 controls are tested: 2C change management, 2D supervision and 2E testing. ISACA does not map tasks or traps to subareas.
Sources and fact check
Source checked: 2026-10-17
- Study methods and examples are independent CertArc teaching guidance. ISACA's pages are the authority for exam rules and fees, which can change.
- Practice results do not convert to an official score or guarantee a pass. CertArc is not affiliated with or endorsed by ISACA.
Frequently asked questions
Is a model card ever enough evidence?
It is enough to understand the model's intended use and limits. It is not enough to conclude that a control operated.
Can I rely on a supplier's assurance report?
Partly, if its scope and period cover the controls you rely on. Read the exceptions, and test the controls your organization runs itself, such as reviewing the supplier's outputs.
What is the best evidence for human oversight of AI?
Records of actual reviews and overrides for a sample of decisions, drawn from a complete population for the audit period.
Are AI system logs always reliable?
No. Check who can change them, whether they are complete and whether they cover the model version in scope.
Which domain does this trap belong to?
Mainly Domain 3, AI Auditing Tools and Techniques, applied to the controls in Domain 2.
How do I practise it?
Before reading the options, write down the record the control should leave. Then check whether your chosen option tests that record.
