Capacity Planning

Applies tov1.4.0HardwareNVIDIA / AMDDifficultyIntermediateImpactNo restart

Capacity planning for LLM serving comes down to one question: after the model weights are resident, how much VRAM is left for the KV cache — and how many concurrent requests does that buy you?

Get this wrong in one direction and you pay for idle GPUs. Get it wrong in the other and you meet ERR_OCULIS_CUDA_OOM in production.

Interactive planner

Serving configuration

Concurrent requests

GPU count

Memory utilization ceiling0.90

Estimated footprint

Calculating…

• Weights
• KV cache
• Overhead
• Headroom

Model weights

—

KV cache (total)

—

KV per request

—

Runtime overhead

—

Total required

—

Usable VRAM

—

—concurrent requests fit at this context length.

Estimates assume paged attention with no prefix-cache sharing, and exclude CUDA graph capture and fragmentation. Treat the result as a starting point, then confirm with a load test.

The arithmetic behind the planner

You should be able to reproduce the number by hand, because the planner cannot know your real traffic shape.

1. Model weights

weights_bytes = parameter_count × bytes_per_parameter

Precision	Bytes per parameter	70B model
FP32	4	263 GB
FP16 / BF16	2	131 GB
FP8	1	66 GB
INT4 (AWQ/GPTQ)	0.5	33 GB

Weights are a fixed cost. They do not vary with load, which makes them the easy part.

2. KV cache

This is the part that scales with traffic, and the part people forget.

kv_bytes_per_token = 2 × layers × kv_heads × head_dim × bytes_per_element
kv_bytes_total     = kv_bytes_per_token × context_length × concurrent_sequences

The leading 2 is for the key tensor and the value tensor. Note that it is kv_heads, not attention heads — grouped-query attention (GQA) models like Llama 3.1 share KV heads across query heads, which is why a 70B model with 8 KV heads is far cheaper to serve at long context than its parameter count suggests.

Why context length dominates

KV cache is linear in context length and linear in concurrency, while weights are constant. At 128K context, a single Llama 3.1 70B request needs roughly 5 GB of KV cache on its own — more than the weights of an 8B model. Long-context serving is a memory-bandwidth problem long before it is a compute problem.

3. Runtime overhead

Budget for activations, the CUDA context, allocator fragmentation, and CUDA graph capture. The planner uses 1.2 GB per GPU + 6% of weights, which is deliberately conservative.

4. Usable VRAM

Never plan against the nameplate number. The serving runtime reserves a fraction, and you want headroom for traffic spikes:

usable = vram_per_gpu × gpu_count × gpu_memory_utilization

gpu_memory_utilization defaults to 0.90 in Oculis. Raising it above 0.95 trades your safety margin for a small concurrency gain and is a common cause of OOM under burst.

Applying the result

1. Set the memory ceiling and context limit to match what you planned for.

   CLI / YAML

   oculis-config.yaml

   oculis:
     upstreams:
       - name: vllm-primary
         provider: vllm
         endpoint: 'http://localhost:8000/v1'
         engine:
           gpu_memory_utilization: 0.90
           max_model_len: 8192
           max_num_seqs: 32
           kv_cache_dtype: fp8 # halves KV cache footprint

   Helm

   values.yaml

   upstreams:
     - name: vllm-primary
       provider: vllm
       endpoint: http://vllm.inference.svc.cluster.local:8000/v1
       engine:
         gpuMemoryUtilization: 0.90
         maxModelLen: 8192
         maxNumSeqs: 32
         kvCacheDtype: fp8
   resources:
     limits:
       nvidia.com/gpu: 1

   Python

   plan.py

   from oculis import UpstreamConfig, EngineConfig
   
   upstream = UpstreamConfig(
       name="vllm-primary",
       provider="vllm",
       endpoint="http://localhost:8000/v1",
       engine=EngineConfig(
           gpu_memory_utilization=0.90,
           max_model_len=8192,
           max_num_seqs=32,
           kv_cache_dtype="fp8",
       ),
   )

2. Confirm the engine agrees with your estimate. The runtime reports its actual block allocation at startup — this is the number that matters, not the planner’s.

   oculis upstream inspect vllm-primary --show-memory

3. Load-test to the ceiling you planned for, following benchmarking methodology. A plan you have not tested is a guess with extra steps.

Sizing rules that hold up in production

• Plan for p99 context, not mean context. Retrieval-augmented traffic has a long right tail, and OOM is triggered by the tail.
• Leave 10–15% headroom. A configuration that exactly fits will OOM the first time traffic bursts or a long document arrives.
• Prefer FP8 KV cache over reducing max_model_len. It roughly doubles concurrency at negligible quality cost on most models. See KV cache quantization.
• Quantize weights before adding GPUs. Going from FP16 to FP8 on a 70B model frees ~65 GB — often an entire card’s worth.
• Watch preemption rate, not just memory. If the engine is preempting and recomputing sequences, you are already over capacity even though nothing has errored yet. That surfaces as ERR_OCULIS_KV_CACHE_EXHAUSTED.

Known limitations of this model

The planner assumes paged attention with no cross-request prefix sharing. Two things will make your real footprint differ:

• Prefix caching reduces KV usage when many requests share a long system prompt — sometimes dramatically. The planner does not model this, so it over-estimates.
• Mixture-of-experts models (Mixtral, DeepSeek-V3) keep all expert weights resident while activating only a subset per token. Size weights by total parameters, not active parameters. The planner already does this for Mixtral 8×7B.