An AI system does not need every byte to sit in its fastest memory at every moment. It needs the model weights, active key-value cache, runtime workspace and current batch to fit or move quickly enough that the required service level still holds. Confusing those requirements can turn an expensive HBM purchase into idle capacity—or turn an apparently cheaper system into a transfer bottleneck.
Public 2026 technical material points in two directions that are easy to mistake for a contradiction. Accelerator and memory suppliers continue to increase on-package capacity and bandwidth, while serving systems increasingly place selected key-value cache blocks in host memory or storage. Both can be rational: the hot tier must satisfy residency and latency constraints, and colder tiers are useful only when reuse repays the movement cost.
The operational decision is therefore not “buy the tallest stack” or “offload everything”. It is: separate capacity, bandwidth, placement and movement, then benchmark the whole memory path with the model and traffic that will actually run.
Separate the public specification from the workload assumption
NVIDIA's July 2026 Rubin architecture account specifies up to 288GB of HBM4 and 22TB/s of memory bandwidth per GPU. NVIDIA explicitly says capacity supports model residency, larger KV caches and concurrency, while bandwidth feeds token-by-token decode. That is not evidence that capacity has stopped mattering; it is evidence that the two dimensions perform different jobs.
Memory suppliers are also still developing tall stacks. SK hynix says its 12-layer HBM4E samples provide 48GB per stack and up to 16Gbps per pin. Samsung's earlier 12-layer HBM3E packaging announcement describes the engineering needed to keep a 12-layer product within the height of an 8-layer package and says its bonding process helps product yield.
Those public facts support a narrower conclusion. Stack height, package geometry, thermal behaviour, speed binning, yield, capacity and price are coupled design variables. They do not verify a particular unreleased SKU, a customer request or a percentage of accelerator cost. Keep a claim register with confirmed, supplier-stated, reported and unknown rather than blending them into one roadmap.
Capacity and bandwidth fail in different ways
Before selecting an accelerator, classify the constraint seen in production traces.
| Constraint | Typical symptom | Evidence to collect | Likely control |
|---|---|---|---|
| Resident capacity | Model or active KV state does not fit; requests are evicted or rejected | Peak bytes by weights, KV cache, workspace and batch | More HBM, smaller model, quantisation or lower concurrency |
| Achieved bandwidth | Decode is slow even though the workload fits | Bytes moved and achieved bandwidth during decode | Faster memory, better kernels, locality or different parallelism |
| Transfer path | Time to first token rises when state moves between tiers or workers | Transfer size, queue time and link utilisation by hop | Keep hot state local, increase link throughput or reduce movement |
| Fragmentation and allocation | Free memory exists but cannot admit the next sequence efficiently | Allocated, reserved and usable blocks over time | Paged allocation, tighter batch policy or cache compaction |
| Reuse failure | Common prefixes are repeatedly recomputed | Prefix hit rate, reuse distance and recompute time | KV-aware routing and an attributable shared cache |
| Thermal or power limit | Clock or throughput falls under sustained load | Rack power, temperature, throttling and task throughput | Power and cooling redesign or a lower-density operating point |
Build a resident-set ledger for each model release. At minimum, include weights at the deployed precision, active KV bytes by context and concurrency, runtime workspace, communication buffers, safety headroom and any duplicated state created by parallelism. The largest advertised context window is not the expected resident set. Our context-window field guide applies the same principle at the application layer: capacity is a boundary, while a working set is a governed selection.
Measure p50 and p95 time to first token, inter-token latency, completed tasks per second, accelerator utilisation, achieved memory bandwidth, cache hit rate, transfer bytes, spill rate, power and full cost per successful task. A system that produces more raw tokens but misses an interactive latency target has not removed the relevant bottleneck.
Treat HBM as the hot tier, not the whole memory system
Modern serving systems can distribute work and state across accelerator memory, host DRAM, local flash and remote storage. That does not make the tiers interchangeable. Each move adds scheduling, bandwidth and latency costs, so only state with enough reuse or a loose enough deadline should travel down the hierarchy.
The Mooncake paper describes a production architecture for Kimi that separates prefill and decode clusters and uses CPU DRAM and SSD capacity as a disaggregated KV cache. Its reported gains are specific to the evaluated platform and workloads, but it establishes that useful long-context serving can be designed around a memory hierarchy rather than one undifferentiated HBM pool.
NVIDIA's current Dynamo KV Block Manager documentation now exposes the same class of design directly: KV blocks can span GPU memory, pinned host memory, SSD and remote storage. The documentation also states the decision condition clearly—offload helps when reuse avoids enough recomputation to outweigh transfer overhead. A cold block with no future reuse only consumes I/O twice.
Use explicit placement classes:
- hot and required now: active weights, active request state and latency-critical KV blocks stay in HBM;
- warm and likely to return: reusable prefixes or paused sessions may move to host memory when measured recovery time fits the service level;
- cold but expensive to recompute: selected cache blocks may move to local or remote storage with retention, privacy and tenant controls; and
- cheap or unsafe to retain: recompute or discard rather than treating every generated cache block as an asset.
Placement metadata must retain tenant, model version, tokenisation, attention layout, expiry and permission scope. A cache hit is invalid if it reuses state from the wrong model revision or authority boundary. The same caution applies to general AI result reuse in our cache authority guide.
Software moves the capacity threshold; it does not abolish it
Two software improvements illustrate why a lower-capacity configuration can be sufficient for one workload without becoming sufficient for all workloads.
The PagedAttention paper treats KV cache in blocks so serving does not require one large contiguous allocation per sequence. Its evaluated vLLM system reduced fragmentation and shared blocks across requests, improving throughput against the paper's baselines. It recovered wasted capacity; it did not increase physical bandwidth or make an oversized resident set fit by magic.
DeepSeek-V2 introduced Multi-head Latent Attention. The model paper reports a 93.3% KV-cache reduction against DeepSeek 67B in its comparison. That is a model-architecture result, not a setting that can be applied unchanged to every deployed model. Weight quantisation, grouped-query attention, prefix caching and speculative decoding have similarly workload-specific effects on memory, quality and compute.
Run quality evaluation again whenever a memory-saving method changes weights, activations, attention state or decoding. Measure factual accuracy, instruction following, long-context retrieval, tool decisions and tail latency on the same release set. Saving HBM while increasing retries, review time or task failure can raise the complete operating cost described in our production AI cost guide.
Benchmark the hierarchy as one system
Do not compare an 8-high rumour with a 12-high product by multiplying package capacity. Compare deployable serving profiles under the same conditions.
- Freeze the model, weights, tokenizer, serving engine, prompt assembly and quality threshold.
- Replay representative context lengths, output lengths, concurrency, prefix reuse and tool pauses rather than one synthetic average.
- Test a high-capacity baseline, a bandwidth-first profile and a tiered profile with host or storage offload.
- Hold the user-facing SLO constant; reject configurations that win throughput by breaking tail latency or answer quality.
- Record cold start, warm cache and cache-miss results separately.
- Inject link congestion, storage slowdown, worker loss and cache corruption so the fallback path is measured.
- Include rack power, network, host memory, storage, licences, idle reserve and engineering labour in cost.
- Repeat after model, kernel, driver and scheduler changes; yesterday's memory optimum is not a permanent property.
The acceptance metric should be cost per successful, SLO-compliant task, not cost per installed gigabyte or peak token. Publish the test manifest beside the result so the board, supplier and engineering team are comparing the same workload.
Work a tiering decision before buying capacity
Consider a hypothetical customer-support assistant that retrieves policy, reasons over a long conversation and calls account tools. Its weekday traffic has high prefix reuse, but active cases pause while a person supplies information. The design team should not assume that every paused KV block deserves permanent HBM residence.
Start with the larger-HBM profile and establish the quality and latency baseline. Then test a smaller hot tier that admits only active requests, with paused and reusable prefixes in host memory. Finally add local flash for selected long-lived prefixes and compare retrieval time with recomputation. Keep tool results and business state in their authoritative stores; a KV cache is an acceleration artefact, not the case record.
The tiered design ships only if it holds the same answer-quality gate, peak concurrency and p95 latency while lowering complete cost. If host transfer dominates, cache hit rates are weak or the model weights leave too little headroom, the larger-HBM configuration remains the correct answer. If pauses and repeated prefixes create strong reuse, tiering may release expensive hot capacity without changing the user experience. The outcome comes from the trace, not the architectural fashion.
Procure options, observability and exit routes
Most application teams do not buy an HBM stack directly. They select a cloud instance, managed endpoint or server platform whose memory design is already packaged. Procurement should therefore preserve the ability to change the serving profile as the model and traffic evolve.
Require suppliers to disclose available device memory, supported precisions and engines, topology, host-memory and storage paths, scale-up and scale-out links, power envelope, isolation model and the metrics exposed to the customer. Separate hardware reservation from usable service capacity. A reserved accelerator that lacks the required model image, network path or power headroom is not ready capacity.
Avoid contracts that make a single maximum-context demonstration the acceptance test. Include production-shaped replay, a latency distribution, cache behaviour, quality thresholds, failover and a price for the complete route. Keep model artefacts, evaluation data, cache policy and routing logic portable enough to retest another provider.
The practical verdict
A workload that fits a smaller hot tier would not prove that HBM's engineering value has collapsed. It would show that one serving profile found a different balance among capacity, bandwidth, yield, thermals, availability and price. Public Rubin and HBM4E material still emphasises both high capacity and high bandwidth, while production serving systems increasingly expose colder memory tiers below HBM.
Trace the workload, size the hot set, price every transfer, validate quality and buy the memory hierarchy that meets the task—not the tallest stack on the roadmap.
