An AI agent can revise a plan, retry a model call or restore an earlier workflow snapshot. It cannot make a customer unread an email, make a supplier forget an order or turn a completed bank transfer back into an uncommitted database row. Calling every recovery path a “rollback” hides that difference.
The design decision is therefore made before the tool call: can this proposed side effect be rejected before commit, compensated after commit, or only repaired after harm has occurred? Give each action a recovery contract that records its commit evidence, compensation, deadline, residual impact and owner. Let an agent cross from proposal into execution only when that contract is complete and the remaining risk fits the approved boundary.
A database rollback stops at its transaction boundary
Inside one database transaction, rollback has a precise meaning. PostgreSQL documents that ROLLBACK discards the updates made by the current transaction. That guarantee does not reach an email provider, payment processor, CRM or supplier API already called over the network. Each external system owns a separate commit boundary.
The original 1987 Sagas paper addressed long-lived work by splitting it into local transactions and pairing completed steps with compensating transactions. Its crucial qualification remains current: compensation undoes an action semantically but does not necessarily restore the database state that existed when the action began, because other work may have happened in between.
An AI workflow adds a probabilistic planner, not a new transaction primitive. Rewinding the conversation may change what the model believes, while every already committed side effect remains real. Model state, workflow state and business state need separate recovery rules.
This complements our guide to duplicate and stale AI-agent actions. Idempotency prevents the same intent from executing twice; compensation decides what should happen after one valid execution must be counteracted. Neither substitutes for the other.
Classify side effects before exposing the tool
Do not wait for an incident to discover what “undo” means. Classify each tool action while designing the workflow.
| Action class | Example | Recovery meaning | [Automation](/services) boundary |
|---|---|---|---|
| Proposal only | Draft a reply or purchase order | Discard the draft | Agent may act freely inside the authorised data boundary |
| Rejectable before commit | Conditional database update | Abort the local transaction or reject a stale version | Agent may submit; system of record enforces the preconditions |
| Compensable | Cancel a reservation, void an unfulfilled order | Run a domain-specific counter-action and preserve both records | Agent may execute only with a tested compensation and deadline |
| Repairable, not reversible | Correct an email or notify an affected customer | Reduce harm without erasing the first effect | Require tighter approval, monitoring and an accountable owner |
| Irreversible or legally consequential | Release funds, publish a filing, delete the only copy | No reliable automated undo | Keep behind fresh human authority or remove from the agent toolset |
The class can change with time. An order may be cancellable until fulfilment begins, a payment authorisation may be voidable before settlement, and a scheduled message may be retractable until dispatch. Store the cut-off as business state rather than burying it in a prompt. A fluent model prediction that an action is “probably still reversible” is not commit evidence.
The class can also depend on scope. Deleting a derived cache entry may be recoverable from its source; deleting the source and every backup is not. A compensation definition must name the exact resource, lifecycle state and authority under which it works.
Write the recovery contract beside the forward action
A tool schema usually describes how to move forward. Add a recovery contract at the same level of control, not in an incident runbook that the executor never checks.
| Contract field | Question it must answer |
|---|---|
| Operation identity | Which single business intent connects every attempt, commit and compensation? |
| Preconditions | Which object version, approval, amount, recipient and policy must still be current? |
| Commit evidence | Which provider reference or resource version proves the action happened? |
| Outcome states | Can the result be not-started, committed, rejected, compensated, repaired or unknown? |
| Compensation | What counter-action is allowed, with which parameters and authority? |
| Compensation window | Until which event or time is the counter-action valid? |
| Residual effect | What remains visible, chargeable, reportable or harmful after compensation? |
| Escalation owner | Who decides when forward recovery and automated compensation are both unsafe? |
Microsoft's current Compensating Transaction pattern says recovery is application-specific, must account for concurrent work and may require human intervention. It also warns that compensation can fail and should be resumable, observable and built from idempotent commands. The AIEngine contract above is our implementation framework for applying those distributed-systems properties to agent actions; it is not a Microsoft specification.
NIST's AI RMF Core adds the governance boundary: roles for human-AI configurations should be defined, risks and potential impacts documented, and responses to high-priority risks planned. For a consequential tool, the escalation owner and residual effect are therefore part of authorisation, not optional operational notes.
Treat an unknown outcome as a state, not an invitation to retry
The most dangerous recovery path begins with a timeout. The agent sends an order, receives no response and assumes failure. The provider may have committed just before the connection broke. A fresh call can create a second order; an immediate compensation can cancel the wrong one if the first reference is unknown.
Use a durable operation ledger with explicit states: not-started, in-progress, definitely-rejected, definitely-committed, compensated, repair-required and outcome-unknown. A timeout enters outcome-unknown until the executor queries the destination by stable operation identity, provider reference or another authoritative key. It does not enter “safe to try again” merely because no success response arrived.
The transactional outbox pattern addresses one related dual-write gap. AWS shows how a business update and its outgoing event can be stored in the same local transaction, then published by a separate process. Its example also notes that at-least-once delivery still requires idempotent consumers. An outbox cannot make an external API call atomic with the database, but it can prove which committed local decisions still need dispatch and stop a crash from silently losing the next step.
Keep the operation ledger small enough to retain safely but complete enough to drive recovery. Link it to the privacy-safe evidence described in our AI incident replay guide, rather than copying full prompts, credentials or sensitive payloads into every recovery record.
Compensation is another production operation that can fail
Compensation needs authentication, idempotency, retries, monitoring and its own terminal evidence. It is not a cleanup callback that can be trusted because the happy-path test invoked it once.
AWS's saga orchestration guidance separates local transactions coordinated by an orchestrator and highlights the trade-offs: compensation and retries add complexity, sagas are eventually consistent, participants must be idempotent, isolation is weaker and observability becomes important. Those properties argue for a deterministic workflow engine or state machine around consequential actions. The model may propose a plan; it should not invent recovery semantics during an incident.
Record the compensation before starting the forward step. Temporal's engineering example on compensating actions demonstrates the crash gap: if the side effect occurs and the process fails before registering its undo, the recovery list is empty. Its safer ordering registers an “if present” compensation first, then performs the action. The general lesson is provider-neutral: durable recovery intent must exist before the uncertain call, and the counter-action must tolerate the possibility that the forward action did not commit.
Do not assume every compensation runs in simple reverse chronology. Microsoft notes that sensitive stores may need to be corrected first and independent compensations may run in parallel. Encode dependencies explicitly. If cancelling delivery must precede refunding stock, the workflow should enforce that order; a model-generated list is not a concurrency control.
Put the point of no return at the end
Sequence a workflow so information gathering, drafting, validation and reversible reservations happen before an irreversible or high-impact commit. Immediately before that boundary:
- re-read authoritative state instead of relying on conversational memory;
- revalidate recipient, amount, policy, object version and operation identity;
- obtain fresh approval at the risk level of the actual action;
- acquire a short-lived credential limited to that destination and action;
- confirm that every earlier step has either succeeded or reached an accepted state; and
- persist the final commit intent and recovery owner.
The validation layers in From Model Output to Safe Action should reject incomplete or unauthorised proposals before this point. The delegated-credential pattern should then prevent a draft-level approval from becoming release authority. Compensation is a backstop for partial failure, not permission to weaken prevention.
If no adequate compensation exists, shrink the action. Let the agent prepare a payment draft but not release funds, create a private publication preview but not publish, or assemble a deletion manifest while a separately authorised service executes it. Reduced autonomy is an engineering result when irreversibility exceeds the approved risk tolerance.
Worked example: an agent prepares a supplier order
Consider a hypothetical procurement workflow; the values and systems are illustrative, not a report of an AIEngine deployment. An agent reads an approved request, drafts a purchase order, reserves £3,200 from a departmental budget, creates an order in a supplier portal and sends the requester a confirmation.
The draft is discardable. The budget reservation is a conditional local transaction with an expiry. The supplier order is compensable only until the supplier accepts it. The confirmation email is repairable but cannot reliably be unread. Payment release is not part of this workflow.
Before execution, the system creates operation identity [tenant](/industries/real-estate)/request/revision/order, stores the approved supplier, items, maximum amount and cancellation deadline, and registers these recovery actions:
- release the budget reservation if it exists and no invoice is attached;
- cancel the supplier order by the original operation identity if it exists and is still cancellable; and
- if confirmation was sent but the order failed, open a correction task with the recipients and original message reference.
The supplier call times out. The workflow records outcome-unknown and queries the portal by operation identity. If the order exists, it stores the supplier reference and continues or compensates according to current policy. If the portal proves no order exists, it may retry with the same identity. If the portal cannot answer, the workflow pauses for its owner instead of creating another order.
Suppose the order exists but its price exceeds the approved amount. The orchestrator first attempts supplier cancellation, then releases the reservation after cancellation is confirmed. If the email already went out, a correction task remains even after both state changes succeed. The terminal state is compensated-with-residual, not the fiction that nothing happened.
Test every partial boundary
Happy-path unit tests establish little about compensation. Inject failure before and after every commit point:
- crash after the destination commits but before the local ledger records success;
- return a timeout after a real commit and require reconciliation rather than a new intent;
- deliver the same forward and compensation commands more than once;
- advance the resource version or pass the compensation deadline before recovery runs;
- fail one compensation while later steps remain pending;
- resume an old worker after another worker completed recovery;
- withdraw approval between proposal and the point of no return; and
- make the destination unable to prove whether the action happened.
The release evidence should show the resulting state, next eligible command, owner and residual effect for each injected failure. A workflow that ends in an explicit manual-repair queue can be safer than one that reports success while state is inconsistent.
Measure recovery as an outcome
| Measure | What it reveals | Review trigger |
|---|---|---|
| Actions with complete recovery contracts | Tool surface covered before execution | Any production side effect without one |
| Outcome-unknown age | Time spent unable to prove commit state | Any breach of the action-specific reconciliation objective |
| Compensation success and duration | Whether backward recovery actually completes | Retries exhausted or worsening tail time |
| Compensated-with-residual count | Harm that rollback language would conceal | Repeated residuals from one action class |
| Manual repair age and recurrence | Operational debt and ownership | Unowned item or repeating cause |
| Irreversible commits by approval type | Whether authority matches impact | Any commit under draft-level or stale approval |
Revisit a recovery contract when the provider, API semantics, cancellation window, workflow order, approval policy or business consequence changes. The final gate is concrete: can the team prove whether the action committed, run an authorised counter-action safely more than once, identify what cannot be undone and route the residue to an owner? If not, keep the agent on the proposal side of the zipper.



