GPU, CUDA & Memory

Applies tov1.4.0HardwareNVIDIA / AMDDifficultyAdvanced

GPU failures split into three families, and telling them apart early saves hours:

1. Memory — the configuration asks for more VRAM than exists. Fixable in config.
2. Environment — drivers, runtimes, and device visibility. Fixable in the image or host.
3. Hardware — the card is failing. Not fixable in software.

Start here:

oculis status --check-deps --verbose
nvidia-smi --query-gpu=index,name,driver_version,compute_cap,memory.used,memory.total --format=csv

Error code	Symptom & log signature	Root cause & first action
CriticalERR_OCULIS_CUDA_OOMHTTP 503	Requests fail intermittently under load; failures cluster on long prompts rather than long generations. torch.OutOfMemoryError: CUDA out of memory. Tried to allocate	Peak prefill activation memory plus the KV cache exceeds free VRAM. Usually caused by `max_model_len` or `max_num_batched_tokens` set higher than the card can hold. FixLower `gpu_memory_utilization` to 0.88, cap `max_num_batched_tokens`, then size the KV cache with the capacity planner. Full procedure
ErrorERR_OCULIS_KV_CACHE_EXHAUSTEDHTTP 503	Throughput collapses at a fixed concurrency ceiling. Requests queue rather than error, until the queue itself overflows. kv_cache: no free blocks (used=100.0%), preempting seq_group	All KV cache blocks are allocated. The engine begins preempting and recomputing sequences, which multiplies effective prefill cost. FixEnable FP8 KV cache quantization to roughly double block capacity, or reduce `max_num_seqs`. Full procedure
CriticalERR_OCULIS_GPU_UNAVAILABLEHTTP 503	Agent starts but reports zero accelerators; all traffic routes to fallback. device_probe: found 0 CUDA devices (CUDA_VISIBLE_DEVICES="")	The container lacks the NVIDIA runtime, or `CUDA_VISIBLE_DEVICES` is empty or masked by the orchestrator. FixConfirm `nvidia-smi` works inside the container and that the pod requests `nvidia.com/gpu`. Full procedure
CriticalERR_OCULIS_DRIVER_MISMATCHHTTP 500	Agent exits during model load, immediately after the CUDA context is created. CUDA error: forward compatibility was attempted on non supported HW	The CUDA runtime bundled in the image is newer than the host kernel driver supports. FixMatch the driver to the runtime using the compatibility matrix, or install the forward-compat package on the host. Full procedure
CriticalERR_OCULIS_NCCL_TIMEOUTHTTP 500	Multi-GPU deployments hang at startup, then all ranks abort together after ~10 minutes. Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE	One rank never reached the collective — commonly a blocked NCCL port, mismatched `NCCL_SOCKET_IFNAME`, or a peer that OOMed first. FixSet `NCCL_DEBUG=INFO`, confirm every rank sees the same interface, and check for an upstream OOM on rank 0. Full procedure
CriticalERR_OCULIS_ECC_UNCORRECTABLEHTTP 500	A single node produces corrupted output or crashes repeatedly; others are healthy. Xid 48: Double Bit ECC Error	Failing GPU memory. This is a hardware fault, not a configuration problem. FixCordon the node immediately, drain traffic, and retire the card. Do not attempt a software workaround. Full procedure
ErrorERR_OCULIS_MODEL_LOAD_FAILEDHTTP 500	Pod restarts in a loop; readiness probe never passes. model_loader: failed to materialize weights from	Missing or partially downloaded weights, an unreadable cache mount, or a quantization format the runtime was not built with. FixVerify the weight checksum and confirm the cache volume is mounted read-write with sufficient free space. Full procedure

CUDA out of memory

Signature: ERR_OCULIS_CUDA_OOM

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.31 GiB.
GPU 0 has a total capacity of 79.15 GiB of which 1.82 GiB is free.

Why it clusters on long prompts

The counter-intuitive part: OOM usually happens during prefill, not decode. Prefill processes the entire prompt at once, so its peak activation memory scales with prompt length. A 32K-token prompt can momentarily need several GB more than a 200-token prompt generating the same output.

This is why OOM appears “random” — it tracks your prompt-length distribution, and it is the p99 prompt that kills you, not the median.

Resolution

1. Find the actual ceiling rather than guessing.

   oculis upstream inspect vllm-primary --show-memory

   Example output

   GPU 0  NVIDIA H100 80GB
     total                 79.15 GiB
     weights               65.20 GiB   (Llama-3.1-70B, FP16)
     kv cache allocated     9.80 GiB   (2,504 blocks × 16 tokens)
     activation peak        3.10 GiB   (observed, max_num_batched_tokens=8192)
     free                   1.05 GiB   ← insufficient headroom

2. Lower the memory ceiling to leave allocator headroom. Counter-intuitively, reducing gpu_memory_utilization often fixes OOM, because it stops the engine from pre-allocating KV blocks it cannot back under peak activation load.

   oculis-config.yaml

   upstreams:
     - name: vllm-primary
       engine:
         gpu_memory_utilization: 0.88 # was 0.95
         max_num_batched_tokens: 8192 # cap prefill batch size
         max_model_len: 8192

3. Enable chunked prefill so a long prompt is processed in slices rather than one allocation.

   engine:
     enable_chunked_prefill: true
     max_num_batched_tokens: 4096

4. Quantize if the weights alone leave no room. FP8 halves weight footprint on Hopper and Ada.

5. Re-plan with the capacity planner using your p99 prompt length, not your mean.

Raising gpu_memory_utilization is almost never the fix

It is the most common first attempt and it usually makes things worse. Higher utilization means more KV blocks pre-allocated, which means less headroom for the activation spike that caused the OOM.

KV cache exhaustion

Signature: ERR_OCULIS_KV_CACHE_EXHAUSTED

kv_cache: no free blocks (used=100.0%), preempting seq_group req_01HQ8Z…

This is not an error so much as a performance cliff. The engine starts preempting sequences and recomputing them later, which multiplies effective prefill cost. Throughput collapses while nothing technically fails.

Diagnose:

curl -s http://localhost:9090/metrics | grep -E 'kv_cache_utilization|preemption'

Expected output

oculis_kv_cache_utilization{upstream="vllm-primary"} 0.998
oculis_preemption_total{upstream="vllm-primary",reason="kv_cache"} 8412

Any sustained preemption_total growth means you are over capacity.

Resolve, in order of preference:

1. FP8 KV cache — roughly doubles capacity at negligible quality cost. See KV cache quantization.
2. Reduce max_num_seqs — fewer concurrent sequences, each guaranteed its blocks.
3. Reduce max_model_len — if your real prompts are far shorter than the configured window.
4. Add capacity — the honest answer when the first three are exhausted.

No CUDA devices detected

Signature: ERR_OCULIS_GPU_UNAVAILABLE

device_probe: found 0 CUDA devices (CUDA_VISIBLE_DEVICES="")

Work outward from the card:

1. Does the host see the GPU?

   nvidia-smi

   If this fails on the host, the problem is the driver, not Oculis.

2. Does the container see it?

   docker exec oculis nvidia-smi

   If the host works but the container does not, the NVIDIA Container Toolkit is missing or --gpus all was omitted.

3. Is the variable masked?

   docker exec oculis printenv CUDA_VISIBLE_DEVICES

   An empty string hides all devices; unset is different from empty.

4. In Kubernetes, is the device actually requested?

   kubectl get pod -n oculis -o jsonpath='{.items[*].spec.containers[*].resources}'

   Without nvidia.com/gpu in limits, the device plugin will not attach a GPU regardless of node labels.

Driver and runtime mismatch

Signature: ERR_OCULIS_DRIVER_MISMATCH

CUDA error: forward compatibility was attempted on non supported HW

The CUDA runtime in your image is newer than the host driver supports.

# What the host driver supports:
nvidia-smi --query-gpu=driver_version --format=csv,noheader
# What the image expects:
docker inspect ghcr.io/selaware/oculis-agent:1.4.0-cuda12.4 \
  --format '{{index .Config.Labels "cuda.version"}}'

Image tag	Minimum driver
1.4.0-cuda12.4	550.54.14
1.4.0-cuda12.1	535.86.05
1.4.0-cuda11.8	520.61.05

Either upgrade the driver or pull an image built against an older runtime. Do not install the forward-compatibility package on a datacenter driver you also use for other workloads — it changes behavior for everything on the host.

NCCL collective timeouts

Signature: ERR_OCULIS_NCCL_TIMEOUT

Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout=600000ms)

Multi-GPU only. One rank never reached the collective, so every other rank waited and then aborted together. The abort message names the rank that timed out, not the rank that caused it — which is why this is misdiagnosed so often.

1. Find the rank that failed first. Sort logs by timestamp across all ranks; the earliest error is the real one. A rank that OOMed at T+0 causes every other rank to time out at T+600s.

2. Turn on NCCL diagnostics.

   upstreams:
     - name: vllm-primary
       engine:
         env:
           NCCL_DEBUG: INFO
           NCCL_DEBUG_SUBSYS: INIT,GRAPH

3. Confirm every rank agrees on the interface. A mismatched NCCL_SOCKET_IFNAME means ranks try to talk over different networks.

   env:
     NCCL_SOCKET_IFNAME: eth0
     NCCL_IB_DISABLE: '0' # set to 1 only if InfiniBand is absent

4. Check peer-to-peer topology.

   nvidia-smi topo -m

   PHB or SYS links between GPUs that should be NV# indicates the cards are not on the expected NVLink domain, which turns a fast collective into a slow one.

5. In Kubernetes, confirm shared memory. The default 64 MB /dev/shm is too small for NCCL.

   volumes:
     - name: dshm
       emptyDir:
         medium: Memory
         sizeLimit: 8Gi

Hardware faults and Xid codes

Signature: ERR_OCULIS_ECC_UNCORRECTABLE

nvidia-smi -q | grep -A 3 'Xid\|ECC Errors'
dmesg -T | grep -i xid

Xid	Meaning	Action
13	Graphics engine exception	Often a software bug; retry once, then escalate
31	GPU memory page fault	Usually a software bug in the kernel
48	Double-bit ECC error	Hardware. Retire the card.
63	ECC page retirement	Monitor; retire if the count grows
74	NVLink error	Reseat or replace; check topology
79	GPU has fallen off the bus	Hardware. Power or PCIe fault.
94	Contained ECC error	Row remapped; monitor remap count

Xid 48 and Xid 79 are not software problems

Cordon the node and drain it. Continuing to serve from a card producing double-bit ECC errors risks silently corrupted output — which is far worse than an outage because nothing errors.

kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

Check remapping health before returning a card to service:

nvidia-smi --query-remapped-rows=gpu_name,remapped_rows.pending,remapped_rows.failure --format=csv

A failure of Yes means the card cannot remap further. Retire it.

Model fails to load

Signature: ERR_OCULIS_MODEL_LOAD_FAILED

model_loader: failed to materialize weights from /models/llama-3.1-70b

Check, in this order:

1. Weight integrity. A partial download is the most common cause.

   oculis model verify --path /models/llama-3.1-70b --checksums

2. Disk space and mount mode. The cache volume must be writable with room for the full model.

   df -h /models && mount | grep /models

3. Quantization support. Loading an FP8 checkpoint on Ampere fails here — FP8 needs compute capability 8.9+.

4. Tokenizer files. A weights directory missing tokenizer.json loads and then fails at first request rather than at startup.

Verification

Run after any GPU-related change. Confirms devices are visible, the driver and runtime agree, memory is allocated as planned, and a real request completes on the GPU path.

oculis status --check-deps --verbose && \
oculis upstream inspect vllm-primary --show-memory && \
oculis gateway test --route default

Expected output

  accelerators    OK   2x NVIDIA H100 80GB (driver 535.129.03, CUDA 12.2)
  upstream        OK   vllm-primary (queue 0/50, kv_cache 41.2%)

GPU 0  NVIDIA H100 80GB
  weights               65.20 GiB
  kv cache allocated     9.80 GiB
  activation peak        3.10 GiB
  free                   1.05 GiB
  headroom               OK (>= 1 GiB)

[PASS]  Route "default" resolved successfully (latency 14ms).

headroom below 1 GiB means you are one long prompt away from ERR_OCULIS_CUDA_OOM, even though nothing has failed yet.

Next steps

Capacity planningSize VRAM and concurrency so OOM stops happening.

Latency tuningKV cache quantization and chunked prefill in detail.

Collecting diagnosticsBundle GPU state and logs for escalation.