Last updated:
AAISM preparation guide
AAISM validation trap: strong accuracy on the wrong test
Direct answer
An accuracy figure above the requirement looks like proof. The trap is accepting it when the test data doesn't match how the model will be used. The stronger AAISM answer asks whether the validation covered the real conditions, population and measure of the intended use, and requires that evidence before acceptance.
ISACA's outline places validation in 3B, the AI life cycle, which names model selection, training and validation.
The question to ask of any figure is: tested on what, under which conditions, against which requirement?
Start AAISM baselineEvidence that fits the use
| Trap | Why it feels right | What the stronger answer does |
|---|---|---|
| Overall accuracy above target | The number beats the requirement | Checks results for the conditions the model will meet |
| Vendor test results | The vendor tested it thoroughly | Requires testing on the organization's own use |
| One metric for every risk | A single score is easy to compare | Picks the measure that reflects the harm that matters |
Original teaching example
A city plans to use a vendor's licence-plate recognition model at night-time toll points. The vendor reports 98% accuracy on daytime images, and the city's requirement is 95%.
98% beats 95%, but on the wrong conditions. The stronger answer requires validation on night-time images from the toll points, against the city's requirement, before acceptance.
Tested on what?
Evidence has to match the intended use.
Check the data, conditions and population behind a figure before comparing it with a requirement.
Averages hide failures
Break results down by the cases that matter.
A model can meet an overall target while failing a group or condition it will often meet in service.
Choose the right measure
The measure should reflect the harm you care about.
Missing a fraud case and wrongly flagging a customer have different costs; a single accuracy figure treats them the same.
Where this appears in the outline
3B, the AI life cycle, with links to 2C for vendor models.
Vendor-supplied results raise the same question as the vendor trap: does the evidence fit your use?
Sources and fact check
Source checked: 2026-10-12
- Study methods and examples are independent CertArc teaching guidance. ISACA's pages are the authority for exam rules and fees, which can change.
- Practice results do not convert to an official score or guarantee a pass. CertArc is not affiliated with or endorsed by ISACA.
Frequently asked questions
Is a high accuracy figure ever enough?
Not on its own. It must come from data and conditions that match the intended use, and results should also be broken down by the cases that matter and use a measure that reflects the harm.
Who should validate a vendor model?
The deploying organization should obtain evidence for its own use, whether it tests itself or requires the vendor to.
What is intended use?
The specific purpose, users, population and conditions the AI system is approved for.
Which domain does this trap belong to?
Mainly Domain 3, AI technologies and controls.
How do I spot it in a question?
Look for a strong figure measured under different conditions from the ones in the scenario.
