An AI service can accept every request and still complete less useful work. As concurrency rises, requests wait behind long prompts, slow generations, retrieval calls and tools. Time to first token stretches, callers time out, retries arrive as new load, and the service spends scarce capacity finishing answers whose users or deadlines have already gone.
The operating decision belongs at the entrance: should this request run now, wait briefly, take an evaluated cheaper route, or be rejected before it consumes the bottleneck? A large queue postpones that decision; it does not answer it.
Use backpressure as a product control. Estimate the work, attach a deadline and priority, admit it against live weighted concurrency, and keep queues short enough that accepted work can still be valuable when it starts. Return an explicit overload signal when it cannot. This article separates published infrastructure guidance from an AIEngine admission framework and worked example; the weights and thresholds below are design examples to calibrate on the real workload, not universal constants.
The queue is already making a product decision
Traditional request counts are a weak proxy for AI load. One request may classify a sentence; another may read a long document, generate thousands of tokens, call retrieval twice and fan out to several tools. Provider limits can also operate on different dimensions. Anthropic's current rate-limit documentation describes separate request, input-token and output-token limits, short-interval enforcement and 429 responses with retry timing. Those are supplier controls, not a promise that every allowed request will meet the application's latency target.
Self-hosted inference has the same shape. NVIDIA's NIM benchmarking guidance says requests wait once concurrency exceeds available batch capacity; throughput tends to saturate while latency continues to rise. The precise knee depends on model, hardware, prompt and output mix. That is why an application cannot copy a requests-per-second number from a benchmark and call it capacity.
Every accepted item implicitly promises that the system expects to finish it before its result becomes stale. Make that promise explicit with an admission record:
- request class, tenant and authenticated priority;
- arrival time, end-to-end deadline and maximum queue age;
- estimated input, output reserve, retrieval and tool work;
- required model, region, data path and fallback eligibility;
- operation identity for any delayed or repeated work; and
- the decision: run, queue, degrade or reject, with its reason.
If the system cannot reconstruct this record, it cannot explain why a low-value batch displaced an urgent interactive request.
Meter the resource that saturates first
Start with load tests that reproduce prompt lengths, output lengths, cache behaviour, retrieval depth, tool fan-out and cancellation patterns. Find the first resource whose pressure causes useful throughput to flatten or tail latency to breach its objective. It may be model concurrency, input or output token quota, accelerator memory, retrieval connections, a database pool, a tool's rate limit or human-review capacity.
Then express admission in work units tied to that bottleneck.
| Control dimension | Useful signal | Failure of a request-count limit |
|---|---|---|
| Weighted in-flight work | Estimated model and downstream occupancy | One long generation consumes the same slot as one short classification |
| Input and output budget | Token estimate plus a conservative output reserve | Large contexts or outputs exhaust quota inside an apparently safe request rate |
| Fan-out budget | Retrieval, reranker and tool calls spawned by one request | One admitted agent task multiplies into many downstream calls |
| Queue age | Time already spent waiting against the deadline | Queue depth can look stable while every item is already too old |
| Tenant share | In-flight work and recent usage by accountable identity | One caller can monopolise a global pool |
| Review capacity | Oldest item and sustainable reviewer throughput | Automated throughput creates an unbounded human bottleneck |
A simple starting estimate is input work + reserved output work + downstream fan-out work. Calibrate the coefficients from measured service time and quota consumption. Reconcile the estimate with actual use after completion so systematic underestimates become visible. Never let a cheap estimate grant tool authority; admission controls resource use, while the safe-action gates still control what may commit.
Put one admission decision before expensive work
Authenticate and validate a small envelope first, then make one cheap decision before parsing large files, embedding documents or invoking a model. The admission controller needs live headroom, recent tail latency, queue age, the request's work estimate and the policies for its class.
| Decision | Use when | Required result |
|---|---|---|
| Run now | Reserved headroom exists and predicted completion fits the deadline | Consume a weighted permit and propagate the deadline downstream |
| Queue briefly | The burst is shorter than the permitted wait and the work remains valuable | Durable or in-memory queue with age limit, cancellation and an owner |
| Degrade | A separately evaluated route can make a narrower promise with less work | Name the mode, reduce scope and preserve its lower authority |
| Reject | No allowed route can finish in time or a protected share is exhausted | Cheap typed response with retry guidance or an asynchronous alternative |
This is not the same as the broader AI degradation ladder. The ladder defines what reduced service may still be safe; admission control decides whether a new item may enter any rung without damaging work already in progress.
Keep permits at each scarce stage rather than only at the public gateway. A gateway limit cannot see that every accepted request fans out to the same retrieval pool. Conversely, an isolated provider limit cannot protect application memory or reviewer queues. Carry the request identity, class, deadline and remaining work budget across those boundaries so a downstream rejection travels back without becoming a generic failure.
Use queues for bursts, not permanent excess demand
A queue is useful when arrivals are temporarily faster than service and delayed completion is acceptable. Microsoft's queue-based load-levelling pattern explains the benefit and the boundary: if average production exceeds consumption, the queue keeps growing and latency rises. It also warns that unbounded consumer scaling can merely move overload to the downstream dependency.
Give every queue three bounds:
- Work bound: maximum weighted work, not only message count.
- Age bound: discard or reroute an item when it cannot finish before its deadline.
- Ownership bound: a tenant, workflow and operation identity for fairness, cancellation and reconciliation.
Interactive work usually needs a very short queue or immediate rejection. Durable asynchronous work can wait longer, but it needs idempotent consumers because common queue delivery is at least once. Preserve the same operation identity through admission, retry and execution using the agent concurrency and idempotency controls. A stale item must not reappear as a fresh duplicate merely because a worker redelivered it.
Priority is not permission to starve everyone else. Reserve a measured share for urgent interactive or control traffic, cap each tenant, and let background work borrow unused capacity only while it can be reclaimed. Authenticate the priority class server-side. A client-supplied “urgent” field should never bypass the share it is entitled to use.
Send overload back to the caller
Backpressure fails when one layer absorbs a downstream overload signal and keeps retrying. Microsoft's updated throttling pattern recommends aligning limits with the component that saturates first, reacting before collapse, using tail latency against the service objective and propagating downstream throttling upstream. That is a stronger trigger than waiting for average utilisation to reach 100%.
Use response semantics deliberately. RFC 6585 defines HTTP 429 for a caller that has sent too many requests in a period and allows Retry-After. RFC 9110 defines 503 for temporary service overload and permits the same timing signal. The distinction helps a caller decide whether to slow one tenant or treat the service as broadly unavailable.
Include the affected scope, a stable reason, whether the work was accepted, and a realistic retry time. Do not return success for dropped work. Do not invite an immediate retry when capacity will not exist. If asynchronous intake is available, return an operation identifier and status route only after the durable queue has accepted the item.
Give retries one owner and one budget
A retry consumes the same constrained path as original work and sometimes more, because the first attempt may still be running. AWS's retry-control guidance calls for one appropriate retry layer, exponential backoff with jitter, maximum attempts, idempotency and explicit testing. gRPC's retry documentation adds a useful mechanism: a token-based throttle pauses retries when failures consume the budget faster than successes replenish it.
Apply those principles to the whole AI workflow:
- choose one component that owns retries for each call;
- count automatic SDK, gateway, workflow and user-interface attempts together;
- retry only errors classified as transient and only inside the original deadline;
- release or cancel the first attempt when the protocol permits;
- carry the same operation identity for state-changing work; and
- stop retries when their recent success yield no longer justifies their load.
Track retry amplification = total attempts divided by original admitted requests by dependency and error class. A falling success rate with rising amplification is a control failure, not extra availability.
Worked design: a policy assistant at the daily peak
Consider a hypothetical policy assistant, not a deployed AIEngine case. Interactive staff questions share one model pool with document ingestion and a nightly evaluation job. The team finds in load tests that long document answers occupy roughly four times the bottleneck service time of short questions, while an evaluation case occupies twice as much and can wait.
The controller assigns illustrative weights of one, four and two. It reserves 70% of measured safe concurrency for interactive work, lets ingestion borrow idle interactive permits, and admits evaluation work only from the remainder. Each question carries an eight-second end-to-end deadline; any predicted wait above two seconds routes to an evaluated concise-answer mode or returns 503 with Retry-After. Ingestion is durable, tenant-capped and expires after its business freshness window. Evaluation pauses first.
| Observed condition | Admission action | Evidence to retain |
|---|---|---|
| Normal tail latency and spare weighted permits | Run the declared route | Estimate, permit class and actual work |
| Short interactive burst; predicted wait below two seconds | Enter the small interactive queue | Arrival, deadline, queue age and cancellation |
| Output-token headroom low; concise mode passes its own gate | Run concise mode with lower output reserve | Mode, disclosure and quality checks |
| Retrieval pool saturated although model permits remain | Reject or defer source-based answers | Downstream bottleneck and retry time |
| Queue age breaches its bound | Expire oldest low-priority work; do not refresh its timestamp | Original operation ID and final disposition |
| Provider recovers after sustained overload | Increase admitted work gradually | Recovery probes, latency and error rate |
The weights are deliberately local. A model change, longer context, different batch engine or new tool can move the bottleneck, so the controller must be recalibrated with the release. The context working-set guide provides the corresponding discipline for reducing admitted context rather than treating the advertised window as free capacity.
Test the overload knee and the recovery path
Google's SRE chapter on cascading failures recommends realistic overload testing, early cheap rejection, small queues, load shedding, deadlines and retry budgets. For an AI path, the test needs to preserve workload variety rather than replaying identical short prompts.
Run a stepped test through and beyond the expected concurrency knee. Mix long and short input, variable output, cache hits and misses, tool fan-out, slow dependencies, cancellations and several tenant shares. Inject 429, 503, timeout and partial-stream conditions separately. Verify that:
- useful completions before deadline plateau rather than collapse;
- new low-priority work is shed before accepted work misses its objective;
- expired and cancelled requests stop consuming downstream capacity;
- retry amplification stays inside budget at every layer;
- protected shares survive a noisy tenant and background backlog;
- the degraded route meets its own quality and authority gate; and
- recovery ramps gradually without rebuilding the queue that caused the incident.
Measure accepted, queued, degraded and rejected work by reason; weighted in-flight work; queue-age percentiles; useful completion before deadline; wasted tokens after cancellation; retry amplification; provider quota headroom; and fairness by tenant and priority. Revisit the controller whenever the model, prompt, output cap, cache, retrieval path, tool graph, hardware, provider limits or review capacity changes.
The release question is concrete: under overload, does the system spend its next unit of capacity on the work most likely to finish usefully before its deadline? If the evidence cannot answer that, shorten the queue, narrow the service promise or reject earlier. Accepted work is a capacity commitment, not an optimistic receipt.
