An AI agent does not need to make two different decisions to create two payments, two emails or two configuration changes. One decision can be executed twice because a request timed out, a queue redelivered a message, two workers claimed the same task or a paused worker resumed after its lease expired.
That is a concurrency-control problem, not a model-quality problem. Better prompting cannot establish whether an external side effect already happened. A human approval does not help if the approved action is replayed twice. Output validation can establish that an action is allowed, but the execution layer must still ensure that the allowed action occurs at most once and that an obsolete worker cannot commit it later.
The design decision is therefore specific: which system [property](/industries/real-estate) must reject a duplicate, conflicting or stale action at the point where it becomes real? Four controls answer different parts of that question—idempotency keys, conditional versions, fencing tokens and serializable transactions. They are complements, not interchangeable labels for “safety”.
Name the failure before choosing the primitive
Start with the observable race rather than the technology already in the stack.
| Failure | What happened | Property required | Primary control |
|---|---|---|---|
| Duplicate retry | The same approved intent arrived again | One result for one operation identity | Idempotency record |
| Lost update | Two workers acted on the same old state | Write only if the version is unchanged | Conditional write or compare-and-swap |
| Stale lease holder | An old worker resumed after a successor took over | Newer authority must supersede older authority | Monotonic fencing token checked by the sink |
| Broken multi-record invariant | Several valid writes formed an invalid combined state | The change must appear in one serial order | Serializable transaction or single-writer queue |
| Out-of-order workflow | A later step arrived before an earlier one | Only valid state transitions may commit | Versioned state machine |
This classification matters because a duplicate key does not detect an outdated decision, and a lease does not prove that its holder is still the only writer. The same separation appears in our guide to authority boundaries for reused AI results: equivalence, freshness and permission are independent questions.
Idempotency gives one intent one durable identity
The HTTP standard defines an idempotent method by its intended server effect: repeating identical requests should have the same intended effect as sending one. It also warns clients not to retry a non-idempotent request automatically unless they know its semantics or can detect that the original was not applied (RFC 9110, section 9.2.2). Agent tool calls are often POST-like business operations, so the application has to supply the missing identity.
An idempotency key should represent one approved business intent, not one HTTP attempt and not merely a hash of the prompt. A useful namespace includes the tenant, target, action and approval or workflow version: for example, a refund operation for one case at one approved revision. Every retry of that operation carries the same key; a genuinely new decision gets a new key.
The service should atomically create or read a record containing at least:
- the idempotency key and tenant boundary;
- a fingerprint of the material request parameters;
- a state such as started, committed, failed-before-commit or outcome-unknown;
- the external provider reference or committed resource version; and
- the response that later callers should receive.
AWS's Builders' Library describes the same client-request-identifier pattern and highlights two traps: the same identifier arriving with different intent, and a request arriving late after the caller has moved on (Making retries safe with idempotent APIs). Parameter fingerprints turn the first trap into a conflict instead of silently returning the wrong result. A retention rule tied to the business retry horizon addresses the second; deleting keys too early reopens the duplicate window.
Stripe's API is a concrete implementation: a caller supplies an idempotency key for a POST, subsequent requests with that key receive the stored status and body, and changed parameters are rejected. Stripe also documents that keys can be removed after at least 24 hours and that validation or concurrent conflicts before execution are not stored (Stripe idempotent requests). Those are provider semantics, not a universal recipe. If an agent may retry a payroll or procurement action after three days, its own operation ledger must outlive the provider's minimum window.
Idempotency also needs an atomic boundary. Writing “started” after calling the payment provider leaves a crash gap in which the payment may succeed but the ledger remains empty. Record the operation before execution, pass the same key downstream where supported, then store the provider reference. If the process dies between the external commit and local confirmation, mark the result outcome-unknown and reconcile by provider reference; do not guess and issue a new operation.
Conditional versions reject decisions made on old state
Idempotency answers “have we already attempted this intent?” It does not answer “was the intent still valid when we tried?” Suppose two agents read an invoice in pending state. One approves it; the other, using the same snapshot, tries to place it on hold. Both operations are individually unique, so two different idempotency keys will not prevent the conflict.
The write must carry the version the decision was based on. In HTTP that can be a strong entity tag sent with If-Match. RFC 9110 explicitly describes If-Match as a way to prevent the lost-update problem when multiple user agents act in parallel (RFC 9110, section 13.1.1). In a database, the equivalent is an update whose predicate includes the expected version and increments it on success.
A version mismatch is not a transient error to retry with the same payload. It means the evidence for the decision is stale. The agent must re-read the current state, recompute the proposed action, pass it through the relevant output validation and action gates, and obtain fresh approval if the policy requires it. Blindly replacing the version with the latest value defeats the control.
Use versions on business objects and workflow transitions, not only on cache entries. A state machine can permit pending-to-approved and pending-to-held while rejecting approved-to-held unless a separate compensating procedure exists. That places the invariant where every writer—agent, human interface, batch job or webhook—must respect it.
A lease proves liveness temporarily, not continuing authority
Leases are valuable for worker coordination. Kubernetes uses Lease objects for node heartbeats and leader election, including to ensure that only one control-plane component is active (Kubernetes Leases). etcd attaches time-to-live leases to keys and exposes revision metadata that clients can use for coordination (etcd API). Neither pattern means a worker becomes harmless the instant its lease expires.
Consider worker A, which acquires a lease and starts a slow tool call. A long pause or network partition prevents renewal. The coordinator expires A's lease, worker B acquires the task and completes it, then A resumes with its old credentials and result. If the destination checks only that A once held a lock, A can still overwrite B.
A fencing token closes that gap. Each ownership grant receives a monotonically increasing number from a linearizable coordinator. Every protected write carries the token. The destination stores the highest token accepted for that resource and rejects any lower token, even if the older worker is alive and believes it still owns the task.
The check has to be performed by the system that owns the side effect. A random lock value can prove identity for safe release, but it cannot order two owners. A timestamp from worker clocks is weaker still. Redis's own distributed-lock documentation now cautions that a process can outlive the lock's validity and recommends implementing fencing tokens, especially for long-running work (Distributed Locks with Redis).
If the destination cannot compare a fencing token—many third-party email, payment and SaaS APIs cannot—do not pretend the lease provides exactly-once execution. Route the action through a database-backed commit ledger, a partitioned single-writer service or a provider idempotency mechanism. Lease loss should also cancel local work, but cancellation is an efficiency measure; sink-side rejection is the authority boundary.
Put multi-record invariants at the system of record
Some actions are safe only as a group. Reserving inventory, recording the order and consuming a budget limit may touch several rows. Per-row versions can all succeed while the combined state violates a cross-row rule.
PostgreSQL's Serializable isolation level guarantees the same effect as if committed transactions ran one at a time in some order, and it requires applications to retry the complete transaction when a serialization anomaly is detected (PostgreSQL transaction isolation). Unique constraints and conflict-aware inserts can also make a business operation key impossible to commit twice. These controls turn a race into a rejected transaction rather than a quietly inconsistent record.
Keep model inference and slow network calls outside a long database transaction. A practical boundary is:
- read a versioned snapshot and prepare the proposed action;
- validate and approve it;
- in a short transaction, claim the operation key, verify the expected versions and reserve the invariant;
- perform the external action with the same idempotency identity; and
- record the external reference or flag an unknown outcome for reconciliation.
External systems prevent a literal single transaction across every step. The operation ledger is therefore not decorative audit data; it is the recovery protocol. It should let an operator distinguish not-started, definitely-committed, definitely-not-committed and unknown. Privacy-safe traces, as described in our AI incident replay guide, should link attempts to that stable operation without storing unnecessary prompt contents.
Worked example: two refund workers, one approved action
Consider a hypothetical support agent asked to execute a refund after a human approves case 1842 at revision seven. A queue redelivery starts workers A and B.
Both derive the same operation identity from the tenant, case, action and approved revision. Worker A atomically creates the operation record; worker B finds the matching fingerprint and waits for or replays A's result. If B presents different amount or destination data under the same identity, the service rejects it as a conflict.
Before execution, A conditionally moves the case from approved revision seven to executing revision eight. If another process already cancelled or modified the case, that transition fails and the refund is not sent. A receives fencing token 481 from the task coordinator and includes it on every write to the internal refund ledger. A also sends the stable operation identity as the payment provider's idempotency key.
If A stalls and B later takes ownership with token 482, the ledger rejects any write from A's token 481. If the provider confirms the refund but A crashes before recording it locally, B queries the provider using the same operation identity rather than creating a new refund. The design uses four boundaries because it faces four questions: duplicate intent, current business state, current worker authority and uncertain external outcome.
Test the crash points and measure the rejections
Happy-path tests will not establish these properties. Inject failures immediately before and after the operation record, conditional transition, external request and local confirmation. Pause a worker beyond its lease, start a successor, then resume the old process. Deliver the same message concurrently and after the normal key-retention window. Reorder workflow events. Force a serialization failure and confirm that the complete transaction—not just the final statement—is retried.
These scenarios belong in release evidence alongside the tests created from red-team findings. NIST's AI Risk Management Framework calls for pre-deployment testing plus ongoing monitoring in production (AI RMF Core); for concurrency controls, production evidence should include both prevented failures and unresolved ambiguity.
| Measure | What it reveals | Review trigger |
|---|---|---|
| Duplicate attempts and replayed results | Retry and redelivery pressure | A sudden source-specific increase |
| Version conflicts by action | Decisions formed on stale state | Repeated conflicts on one workflow step |
| Rejected fencing tokens | Old workers reaching the sink | Any sustained rate above tested failover behaviour |
| Serialization retries | Contention around shared invariants | Rising latency or exhausted retry budgets |
| Outcome-unknown operations | External commits without local proof | Any item beyond the reconciliation objective |
| Side effects per operation identity | The end-to-end safety property | Any value greater than one |
The final review question is not “do we have a lock?” It is whether every side effect has a durable identity, every state-dependent decision names the version it saw, every replaceable worker carries ordered authority, and every shared invariant is rejected atomically at its owner. If one of those checks is absent, document the resulting delivery guarantee honestly and keep the action inside a smaller permission or value boundary until the evidence improves.
