Sampling AI outputs and decisions
Selecting AI decisions or outputs to test.
Auditors sample decisions to re-perform or review, often stratified by risk, group or confidence score, and test edge cases and overrides. For generative systems, repeated prompts show how much outputs vary.
On the exam
Stratify samples, for example by demographic group or by low-confidence scores, to find where errors and bias concentrate.
Related terms
- Sampling (statistical and judgemental)Testing part of a population to conclude about the whole.
- Fairness metricsMeasures for comparing a model’s outcomes across groups.
- Evaluating generative AI (evals)Testing generative AI systems for quality, accuracy and safety.
AAIA Test yourself on this with timed AAIA practice questions in the AAIA Exam Simulation Hub, or read the free AAIA exam guide.
Test yourself with today’s question of the day.
