AI Evaluation Sample Size: Prove the Risk You Claim

An AI evaluation can run 100 cases, find no critical failures and still provide weak evidence. Zero observed failures is not a zero failure rate. A two-point improvement may be real, or it may be smaller than the test set can resolve. An overall score can rise while a rare, high-impact workflow becomes worse.

The missing decision is not “how many examples can we afford?” It is what claim must this evaluation support, with what uncertainty, about which population of work? Only then can a team choose the cases, repetitions and analysis that make the release gate honest.

This is a measurement problem as much as an AI problem. The NIST AI Risk Management Framework calls for documented test sets and methods, representative evaluation where people are involved, performance demonstrated under deployment-like conditions, and limitations on generalisability to be recorded. A raw pass rate without a population, interval or resolution does not meet that standard of evidence.

Start with the release claim, not a round case count

Different claims require different evaluation designs. A single headline accuracy cannot answer all of them.

Write the consequence first. A one-point change in email tone may not alter a launch decision; one unauthorised payment instruction should. The first can use an estimation target. The second needs a strict risk ceiling, deterministic controls outside the model and a test designed around the remaining failure mode.

OpenAI's evaluation guidance usefully separates early error discovery from later measurement. It recommends reviewing roughly 50 to 100 early outputs to build an error taxonomy, then testing under real-world conditions and including rare but costly edge cases (OpenAI's contextual eval framework). Those first 100 examples are a discovery tool. They are not automatically a statistically defensible release certificate.

Define the population before sampling it

An interval describes uncertainty about the population from which the cases could reasonably have been sampled. If a support assistant will receive English and Arabic messages, short questions and multi-document disputes, clean text and OCR noise, then 500 hand-written English questions are not a sample of its deployment population merely because the JSON schema matches.

Record a sampling frame with the dimensions that can change the outcome:

• workflow, user group and language;
• input source, length, quality and attachment pattern;
• tool availability, permission tier and policy version;
• known risk family and severity; and
• cluster identity, such as template, customer, document or conversation.

Anthropic's evaluation guidance similarly says task-specific evals should mirror the real task distribution and include edge cases (Claude evaluation design principles). The practical addition is to name what “mirror” means numerically. A natural-frequency sample estimates ordinary traffic. A risk-weighted sample deliberately over-represents costly cases. Both are useful, but their aggregate scores answer different questions and must not be compared without the sampling weights.

A test row is also not always an independent item. Ten paraphrases of one template share difficulty. Five turns from one conversation share context. Running the same prompt across five random seeds measures generation variability, but it does not create five independent examples of customer need. Keep item count, cluster count, generations per item and graded observations as separate fields.

Zero failures still has an upper bound

For a binary critical failure, model each genuinely independent case as a Bernoulli trial only when the cases are representative of the claim. With zero failures in *n* trials, the exact one-sided 95% upper bound is:

p_upper = 1 − 0.05^(1/n)

This is the zero-failure form of the exact binomial method. NIST explains why normal, symmetric intervals can be inaccurate for small samples or very few failures and documents the exact binomial construction (NIST confidence intervals for proportions).

These figures are AIEngine calculations from the binomial model, not universal compliance thresholds. They assume a stable failure definition, independent observations and a representative sampling process. If 598 rows are produced from 60 near-duplicate templates, the calculation is overconfident. If the grader misses critical failures, the bound is about grader-labelled failures, not real failures.

The formula also exposes an important tradeoff. A severe risk target can demand thousands of cases even before slices are considered. When that is unaffordable, reduce the action's authority, add a deterministic barrier, require human approval or state that the evaluation cannot support the desired ceiling. Do not solve a measurement shortfall by changing “95%” to “high confidence” in prose.

Size comparisons around the smallest decision-changing gap

An upgrade question is different: can the suite detect a regression or improvement large enough to matter? Choose that minimum effect before seeing results. NIST's sample-size method for proportions makes the required inputs explicit: significance level, power and the change in proportion worth detecting (NIST sample sizes for testing proportions).

Run baseline and candidate on the same items. The paired design removes much of the variation caused by some cases simply being harder than others. For binary outcomes, the informative cases are those on which exactly one system passes. A pilot should therefore estimate the disagreement rate, not only each system's accuracy.

Recent analytical work on LLM benchmark sample complexity derives the same practical result: report intervals, pair comparisons, pilot disagreement, account for clustered items and publish the suite's minimum detectable effect (TMLS sample-complexity report). The authors are explicit that their contribution is analytical rather than an experiment showing better organisational decisions, so treat its sizing formula as a planning model whose assumptions must be checked.

For a two-sided 5% test with 80% power, a useful paired approximation is:

n ≈ 7.85 × q / δ²

Here *q* is the baseline-candidate disagreement rate and *δ* is the smallest absolute pass-rate gap that matters. If a pilot finds 12% disagreement and the release must detect a two-percentage-point change, the approximation requires about 2,355 independent paired items. On only 300 items, the minimum detectable effect is about 5.6 points. A measured two-point improvement on that suite is a direction for more evidence, not proof of improvement.

This is not a new concern created by current models. A peer-reviewed EMNLP study found underpowered experiments common in NLP and estimated that a typical 2,000-sentence machine-translation test had roughly 75% power to detect a one-BLEU-point difference (Card and colleagues, 2020). The durable lesson is to plan power before running the expensive comparison, not explain weak resolution after the preferred candidate wins.

Give important slices their own denominator

Suppose 5% of production requests contain low-quality scans, but failures on those scans can send a case to the wrong queue. A natural 600-case sample would contain about 30 such requests on average. Even a perfect slice result would provide little evidence about a low failure ceiling.

Use two views instead:

• a distribution-weighted sample for overall product performance; and
• predeclared, deliberately enlarged safety and quality slices for release gates.

Report both. Apply weights only when estimating the real traffic aggregate. Do not weight away a severe slice failure because the slice is uncommon. If several language, document and permission slices each need their own 95% claim, plan for multiplicity rather than quietly selecting the best-looking intervals after the run.

This separation complements a blind release-test corpus: blindness protects the validity of the cases, while sample design determines the precision of the claim. It also depends on a calibrated LLM judge when scoring cannot be deterministic. A large, clean suite scored by a biased grader can be precisely wrong.

Build a portfolio of evals, not one giant score

One suite rarely serves discovery, regression detection and risk certification efficiently. Use layers with explicit jobs:

• Sentinels: small deterministic cases for known severe failures; any recurrence blocks release. Convert serious red-team findings into permanent release tests.
• Paired regression suite: stable cases run against baseline and candidate, sized to a declared minimum detectable effect.
• Risk-ceiling slices: independent cases for critical failure families, sized to the upper bound the owner is willing to claim.
• Exploration sample: fresh production-shaped cases used to discover drift and new error classes, then promoted into controlled suites through review.

For each run, preserve the population definition, sampling method, cluster ID, system and grader versions, random seed, per-case baseline and candidate outcomes, adjudications, exclusions, interval method and achieved resolution. That record belongs beside the rollback and approval evidence in the AI model change-control baseline.

Make the release record state what remains unknown

Before approving a change, the release owner should be able to answer:

• Which decision-changing failure or gap was the suite sized to detect?
• What population and slices can the result generalise to?
• How many independent items and clusters were tested?
• What confidence bound or minimum detectable effect accompanies the score?
• Was the comparison paired, and what disagreement did the pilot observe?
• How was the grader calibrated on the same difficult slices?
• Which risks remain below the suite's resolution or outside its sampling frame?

Recalculate when traffic mix, tools, policy, prompt, grader or model behaviour changes. Production incidents and appeals should update the error taxonomy and sampling frame, but not be copied directly into a test set without leakage controls.

The final gate is simple to phrase and demanding to satisfy: does the evidence resolve the risk or regression that would change this release decision? If the answer is no, gather more independent evidence, narrow the claim or reduce the system's authority. A precise score is not the same as a supported decision.