Latency Tuning

Applies tov1.4.0HardwareNVIDIA A100 / H100 / L40SRequiresCUDA 12.1+DifficultyAdvancedImpactRolling restart

Latency has two independent halves, and they respond to different fixes:

• TTFT (time to first token) — queueing plus prefill. This is what users perceive as responsiveness.
• TPOT (time per output token) — decode. Bound by memory bandwidth, not compute.

Measure which one is your problem before changing anything. Tuning TPOT when your problem is queueing wastes a week.

Prerequisites

• Oculis Collector v1.4.0 or later
• A self-hosted runtime you control — most of this page does not apply to a cloud API
• NVIDIA A100/H100/L40S with CUDA 12.1+; FP8 needs compute capability 8.9+
• A benchmark baseline — you cannot claim an improvement without a before
• Telemetry configured so per-hop timings are visible
• Permissions: rolling restart of the inference tier

Verify all of the aboveoculis status --check-deps

Step 0 — Find out which half is slow

oculis trace get <request-id> --format tree

oculis.request                          1,847ms
├── oculis.retrieval                        94ms
├── oculis.queue.wait      depth=12        412ms   ← 22% of total
└── oculis.upstream                      1,290ms
    ├── oculis.prefill     tokens=3891     380ms   ← TTFT
    └── oculis.decode      tokens=512      910ms   ← TPOT

Dominant span	Real problem	Fix on this page
oculis.queue.wait	Saturation	Capacity — see below, then capacity planning
oculis.prefill	Long prompts	Prefix caching, chunked prefill
oculis.decode	Memory bandwidth	KV cache quantization, speculative decoding
oculis.retrieval	Vector store	Vector databases

If queueing dominates, stop reading and add capacity

No amount of kernel tuning fixes an under-provisioned tier. Queue wait above ~20% of total latency means you are over capacity — go to capacity planning. Engine tuning below will produce small gains that saturation immediately eats.

Reducing time to first token

Prompt prefix caching

The highest-leverage setting on this page, and the one most often left off.

RAG and agent workloads send a long, identical system prompt on every request. Without prefix caching, that prefix is re-processed from scratch each time. With it, the KV cache for the shared prefix is computed once and reused.

CLI / YAML

oculis-config.yaml

oculis:
  upstreams:
    - name: vllm-primary
      engine:
        enable_prefix_caching: true
        prefix_cache_blocks: 4096 # KV blocks reserved for shared prefixes
        prefix_cache_min_tokens: 256 # do not cache trivially short prefixes
        prefix_cache_eviction: lru

Helm

values.yaml

upstreams:
  - name: vllm-primary
    engine:
      enablePrefixCaching: true
      prefixCacheBlocks: 4096
      prefixCacheMinTokens: 256
      prefixCacheEviction: lru

Python

tuning.py

from oculis import EngineConfig

config.upstreams[0].engine = EngineConfig(
    enable_prefix_caching=True,
    prefix_cache_blocks=4096,
    prefix_cache_min_tokens=256,
    prefix_cache_eviction="lru",
)

Confirm it is actually hitting:

curl -s http://localhost:9090/metrics | grep prefix_cache

Expected output

oculis_prefix_cache_hit_ratio{upstream="vllm-primary"} 0.84
oculis_prefix_cache_tokens_saved_total{upstream="vllm-primary"} 148204882

Prefix caching requires a stable prefix

The cache keys on an exact token prefix. Putting a timestamp, a request ID, or a per-user name at the start of your system prompt reduces the hit ratio to zero. Order your prompt stable-first, variable-last: system instructions, then retrieved context, then the user turn.

Chunked prefill

Long prefills block the decode loop, so one 32K-token request stalls every streaming response on the node. Chunked prefill interleaves prefill work with decode.

oculis-config.yaml

upstreams:
  - name: vllm-primary
    engine:
      enable_chunked_prefill: true
      max_num_batched_tokens: 4096 # prefill chunk size

The trade-off is direct: smaller chunks mean smoother TPOT for everyone and slightly slower TTFT for the long request. Interactive workloads should prefer smaller chunks.

max_num_batched_tokens	Effect
2048	Smoothest inter-token latency; slowest long prefills
4096	Balanced. Reasonable default for mixed traffic.
8192+	Fastest long prefills; visible stutter for streaming users

Priority scheduling

Stop batch traffic from queueing ahead of interactive traffic:

oculis:
  routes:
    - name: interactive
      priority: high
    - name: batch
      priority: low
      max_queue_wait_ms: 30000 # batch may wait; users may not

Reducing time per output token

KV cache quantization

Decode is memory-bandwidth bound. Halving the KV cache roughly doubles how much fits in cache and reduces the bandwidth cost of every decode step.

oculis-config.yaml

oculis:
  upstreams:
    - name: vllm-primary
      engine:
        kv_cache_dtype: fp8 # auto | fp8 | fp8_e5m2 | fp16
        calculate_kv_scales: true

dtype	Memory	Quality impact	Hardware
fp16	Baseline	None	All
fp8	−50%	Negligible on most models	Hopper, Ada (CC 8.9+)
fp8_e5m2	−50%	Slightly worse than fp8	Hopper, Ada; wider range

FP8 requires compute capability 8.9 or newer

On Ampere (A100, A10) and older, kv_cache_dtype: fp8 either silently falls back to FP16 or fails to load the model — ERR_OCULIS_MODEL_LOAD_FAILED. Do not use FP8 quantization on Turing or Volta.

This is also the primary remedy for ERR_OCULIS_KV_CACHE_EXHAUSTED: doubling effective cache capacity is usually cheaper than adding a GPU.

FlashAttention

upstreams:
  - name: vllm-primary
    engine:
      attention_backend: flash_attn # flash_attn | flashinfer | xformers | torch_sdpa

Backend	Best for	Requires
flash_attn	General purpose. Recommended default.	Ampere+
flashinfer	Long context with heavy prefix-cache reuse	Hopper, Ada
xformers	Older hardware without FlashAttention support	Volta+
torch_sdpa	Fallback. Slowest; use for correctness debugging	Any

FlashInfer typically beats FlashAttention on long-context workloads with high prefix-cache hit ratios, and loses on short prompts. Benchmark with your own traffic — the crossover depends on your prompt-length distribution.

CUDA graphs

Eliminates per-step kernel launch overhead, which matters most for small models where launch cost is a large fraction of step time.

engine:
  enforce_eager: false # false enables CUDA graph capture
  cuda_graph_max_batch_size: 256

Capture costs VRAM (typically 1–3 GB) and adds startup time. If you are within a gigabyte of your memory ceiling, this is a plausible OOM cause — see capacity planning.

A tuned starting configuration

For interactive RAG traffic on H100s, this is a reasonable starting point — not a final answer:

oculis-config.yaml

oculis:
  upstreams:
    - name: vllm-primary
      provider: vllm
      endpoint: 'http://localhost:8000/v1'
      model: meta-llama/Llama-3.1-70B-Instruct
      max_queue_depth: 50
      timeout_seconds: 30
      engine:
        # Memory
        gpu_memory_utilization: 0.90
        max_model_len: 8192
        max_num_seqs: 64
        kv_cache_dtype: fp8

        # TTFT
        enable_prefix_caching: true
        prefix_cache_blocks: 4096
        enable_chunked_prefill: true
        max_num_batched_tokens: 4096

        # TPOT
        attention_backend: flash_attn
        enforce_eager: false
        cuda_graph_max_batch_size: 256

  routes:
    - name: interactive
      strategy: least_queue
      upstreams: [vllm-primary]
      priority: high

Verification

Compares against the baseline you captured before tuning. A tuning change without a before-and-after is a guess — the benchmark is the deliverable, not the config.

1. Re-run the identical benchmark you used for the baseline.

oculis bench run \
  --route interactive \
  --profile rag-interactive \
  --duration 300s \
  --compare baseline.json

Expected output

                        baseline      current     delta
  ttft p50                842ms        291ms    -65.4%
  ttft p95              2,841ms        712ms    -74.9%
  tpot p50                 24ms         18ms    -25.0%
  throughput            412 tok/s    689 tok/s   +67.2%
  prefix cache hit          n/a         0.84
  kv cache util            0.99         0.61
  preemptions            8,412            0
  errors                      0            0

[PASS]  No regression in error rate. TTFT p95 improved 74.9%.

2. Confirm the mechanisms you enabled are actually engaging.

curl -s http://localhost:9090/metrics | grep -E 'prefix_cache_hit_ratio|kv_cache_utilization|preemption_total'

Expected output

oculis_prefix_cache_hit_ratio{upstream="vllm-primary"} 0.84
oculis_kv_cache_utilization{upstream="vllm-primary"} 0.61
oculis_preemption_total{upstream="vllm-primary",reason="kv_cache"} 0

A hit ratio near 0 means prefix caching is enabled but not matching — check that your prompt prefix is genuinely stable.

3. Confirm output quality did not move. Quantization is not free, and TTFT is not the only thing that matters:

oculis eval run --route interactive --suite eval/regression.jsonl --compare baseline-eval.json

Edge cases and known limitations

• Prefix caching and per-user prompts do not mix. A user name at the start of the system prompt gives every user their own cache entry and a hit ratio near zero.
• FP8 KV cache changes numerics. The effect is small on most models but not zero. Run an evaluation suite before and after, particularly for structured output and tool calls.
• CUDA graphs consume VRAM you may have already allocated to KV cache. Re-plan capacity after enabling.
• least_queue adds a small routing cost — it inspects upstream state per request. Negligible next to inference, but measurable at very high RPS with tiny prompts.
• None of this applies to cloud providers. Engine settings only affect runtimes you host. For cloud upstreams, latency work means routing and caching.

Next steps

Throughput & costBatching, speculative decoding, and semantic routing.

Capacity planningRe-plan VRAM after changing quantization or enabling CUDA graphs.

Benchmarking methodologyHow to produce a baseline that means something.