AI Abstention: Turn Confidence Into a Routing Policy

An AI system returns a confidence score of 0.93. That number may describe a classifier's largest probability, the likelihood of the next token, a model's self-assessment, agreement across several generations or a separate detector's output. None of those meanings establishes a 93% chance that the complete business outcome is correct.

The production decision is not whether to display the number. It is which route the system may take when evidence is strong, weak, contradictory or outside its tested scope. A useful policy can answer, retrieve more evidence, defer to a competent reviewer or stop into a conventional fallback. It must never turn confidence directly into authority to send, approve, pay or deny.

Build abstention as a measured operating behaviour: define correctness at the task boundary, validate uncertainty signals on representative labelled work, choose thresholds from the consequence of false acceptance and the capacity of the fallback, then monitor risk and coverage by slice. The research below establishes what several signals can and cannot show. The four-route policy, worked design and operating measures are AIEngine's engineering synthesis, not a claim that one score works across models or use cases.

Confidence is a measurement, not permission

The NIST AI RMF Core calls for documenting an AI system's knowledge limits, how people will oversee its output, performance under conditions similar to deployment and the ability to fail safely beyond those limits. It also connects expected error costs to organisational risk tolerance. A threshold should therefore implement a documented decision boundary; it should not manufacture one from the model's preferred scale.

Start by naming the signal precisely.

Keep raw signals, their version and the route decision separate in the trace. That distinction lets a team replace a detector or recalibrate a threshold without pretending the underlying model output changed.

Calibrate the outcome you will actually route

Calibration asks whether events assigned a probability near 0.8 are correct roughly 80% of the time in the defined population. Guo and colleagues' 2017 neural-network calibration study found modern classifiers in its image and document experiments were poorly calibrated and that temperature scaling was often effective there. That is evidence for testing and post-hoc calibration, not permission to apply one temperature to free-form answers or a different deployment.

For a support assistant, label the complete routed outcome: correct policy version, supported material claims, appropriate action, no missing exception and no prohibited disclosure. A fluent answer with a likely opening sentence still fails if its cited policy was superseded. For a tool-using agent, a plausible plan still fails if the destination or authority is wrong.

Use three disjoint datasets:

• a development set for prompts, retrieval and model changes;
• a calibration set for mapping signals to observed outcomes and proposing thresholds; and
• an untouched acceptance set for estimating the final policy's false accepts, abstentions and slice behaviour.

Size that acceptance set around the release claim, not a round case count, using the approach in the AI evaluation sample-size guide. Recalibrate after changing the model, prompt, evidence format, retrieval system, detector, user population or label definition. A probability is calibrated only with respect to a particular outcome and distribution.

Choose a risk-coverage policy before a threshold

Selective prediction makes the trade-off explicit. The 2019 SelectiveNet paper studies a learned reject option and reports improved risk-coverage trade-offs in its classification and regression experiments. Coverage is the share of eligible cases the system accepts; selective risk is the error rate among those accepted. Removing the hardest cases may lower accepted risk, but it also moves work and consequence elsewhere.

Plot risk against coverage on representative work, overall and for material slices. Then choose operating points by route:

One global cutoff is usually too crude. A low-consequence draft may use a broad proposal route while a payment, eligibility decision or customer notification always requires independent validation or approval. The five safe-action gates remain binding even when a confidence signal is high.

Put deterministic evidence ahead of uncertainty

Uncertainty methods help most when the answer could reasonably vary. They should not replace controls that can establish a fact directly. Check identity, permissions, schema, source version, arithmetic, destination, state and commit evidence with authoritative systems before asking a model how confident it feels.

For source-based answers, measure whether each material claim is supported by an approved passage. For extraction, compare totals and identifiers with the original record. For tools, validate live business preconditions and authorisation at execution. A high detector score cannot repair a missing source; a low score should not block a result whose correctness is deterministically established.

This ordering also makes abstention useful. Return a typed reason such as “missing source”, “conflicting policy”, “out of scope”, “uncertain class”, “approval required” or “fallback unavailable”. Typed reasons can trigger retrieval, specialist routing or a safe stop. A generic “low confidence” queue gives reviewers no clue what evidence to seek and makes threshold failures difficult to diagnose.

Treat free-form generation as a signal bundle

Language models can emit confidence-like judgements, but their reliability is conditional. Kadavath and colleagues' self-evaluation study found encouraging calibration for several multiple-choice, true-or-false and self-evaluation formats. It also found that a trained “probability I know” signal generalised only partially and was poorly calibrated on new task distributions. Prompt format and task shift are therefore part of the measurement package.

Repeated generations offer another signal. The 2024 Nature paper on semantic entropy groups answers by meaning and uses disagreement to detect confabulations, a subset of arbitrary, incorrect generations. The authors explicitly distinguish these from systematic errors and say the method does not guarantee factuality. Stable repetition can still reproduce a misconception, bad source or flawed reasoning.

For free-form work, combine signals that fail differently:

• deterministic evidence and policy checks;
• retrieval coverage, freshness and contradiction flags;
• calibrated task-level outcome prediction;
• semantic disagreement or independent sampling where its compute cost is justified;
• a separately validated grader for criteria that require judgement; and
• mandatory review rules based on consequence, regardless of score.

If an automated grader contributes to the route, version and validate it using the LLM-as-a-judge calibration framework. Two uncalibrated models agreeing is not independent assurance.

Use conformal sets for the guarantee they actually make

Conformal prediction can turn model scores into label sets or intervals with an explicit coverage target. Angelopoulos and Bates' conformal-prediction guide describes marginal coverage for a fresh point from the same distribution as the calibration data. It also shows why a basic method can under-cover hard subgroups while over-covering easy ones.

That boundary matters operationally. A 90% marginal coverage target does not mean this prediction is 90% correct, each customer group receives 90% coverage, or the guarantee survives distribution shift. Validate set size and coverage by important slice, watch exchangeability assumptions, and recalibrate when the population changes.

Use set-valued output when the product can act on it honestly: several possible document classes can enter review, or an interval can drive a conservative planning rule. Do not wrap an unconstrained essay in a conformal label and market the result as guaranteed. The output space, score, calibration sample and loss all have to match the decision.

Worked design: route a policy-support answer

Consider a design exercise, not a reported deployment. An internal assistant answers questions from versioned product and refund policies. Every material claim must cite an effective passage; customer-specific exceptions require a case lookup; credits require approval.

• The system first resolves the product, jurisdiction, customer state and effective date. Missing identity or scope stops the route.
• Retrieval must return the applicable policy version and enough passages to support every material claim. Missing evidence triggers one bounded retrieval retry.
• Deterministic checks reject expired policy, contradictory effective dates, unsupported amounts and any proposed credit without approval.
• A calibrated outcome model estimates whether the sourced draft meets the complete label definition. The score chooses between proposal and review; it never approves the credit.
• High semantic disagreement, conflicting sources or an uncovered slice sends the case to a policy specialist with passages, checks and route reasons attached.
• If review wait time exceeds its safe limit, the service returns the conventional callback or case-creation path. It does not widen automation to clear the queue.

Choose the proposal threshold on the calibration set, freeze it, then estimate critical false accepts and coverage on the acceptance set. Report the result by policy family, language, evidence quality and exception type. A blended number can hide that the route is safe for simple returns and unreliable for warranty exclusions.

Make review capacity part of the safety case

Abstention transfers work; it does not remove it. Forecast arrival rate by route reason, handling time, specialist skill, deadline and peak pattern. Reserve capacity for high-consequence cases and define what lower-priority work does when the queue is full. Reviewers need the source evidence, failed checks and uncertainty reason, plus authority to correct the system and record a new label.

Do not tune solely for maximum automation. A threshold that sends half the work to an unstaffed queue is not conservative; it is an unavailable control. Equally, lowering the threshold during a spike converts capacity pressure into hidden false acceptance. Degrade along the pre-agreed route described in the AI failure-ladder guide: narrower scope, slower service or conventional handling before unsafe completion.

Sample accepted, deferred and abandoned work. Reviewers will otherwise see only difficult cases, while silent false accepts remain invisible and needless abstentions are never improved.

Measure whether abstention improves outcomes

Track the policy as an operating system, not a confidence dashboard:

• selective risk and coverage with confidence intervals, overall and by material slice;
• critical false accepts, including cases that should always have stopped;
• false abstentions where deterministic evidence established a safe answer;
• route volume and precision by reason code;
• retrieval salvage rate after a missing-evidence route;
• review wait, abandonment, overturn and correction rates;
• fallback availability and unsafe threshold-override attempts; and
• calibration drift after model, prompt, data or workflow changes.

Investigate when accepted risk rises, a slice loses coverage, review capacity breaches its limit, route reasons change sharply or a mandatory control is bypassed. Recalibration is appropriate when the signal-to-outcome relationship moves; it is not a substitute for repairing bad evidence, labels or permissions.

The final release question is concrete: for every output, can the system explain why it answered, sought evidence, deferred or stopped—and does observed risk support that route? If the answer is no, keep confidence as telemetry. Do not let it become authority.