Difficulty Beginner
Every failure Oculis can report has a stable code. Codes never change meaning across minor
versions, so a code in a two-year-old incident report still resolves to the same entry here.
Each entry is individually addressable. Copy the link from any heading to point a colleague at one
exact failure mode rather than at this page.
Errors are emitted in a consistent shape across logs, API responses, and CLI output:
"code" : "ERR_OCULIS_QUEUE_SATURATED" ,
"message" : "Local queue depth 51 exceeds max_queue_depth=50" ,
"upstream" : "vllm-primary" ,
"request_id" : "req_01HQ8Z3K4M5N6P7Q8R9S" ,
"docs" : "https://docs.selaware.ai/oculis/reference/error-index/#err-oculis-queue-saturated"
Three fields are worth knowing:
request_idoculis trace get <request-id> for the per-hop timing breakdown.retryablefalse; the outcome will not change.docs
GPU & Memory7 Symptom Requests fail intermittently under load; failures cluster on long prompts rather than long generations. Log signature torch.OutOfMemoryError: CUDA out of memory. Tried to allocateRoot cause Peak prefill activation memory plus the KV cache exceeds free VRAM. Usually caused by `max_model_len` or `max_num_batched_tokens` set higher than the card can hold. Resolution Lower `gpu_memory_utilization` to 0.88, cap `max_num_batched_tokens`, then size the KV cache with the capacity planner. Full procedure → Symptom Agent exits during model load, immediately after the CUDA context is created. Log signature CUDA error: forward compatibility was attempted on non supported HWRoot cause The CUDA runtime bundled in the image is newer than the host kernel driver supports. Resolution Match the driver to the runtime using the compatibility matrix, or install the forward-compat package on the host. Full procedure → Symptom A single node produces corrupted output or crashes repeatedly; others are healthy. Log signature Xid 48: Double Bit ECC ErrorRoot cause Failing GPU memory. This is a hardware fault, not a configuration problem. Resolution Cordon the node immediately, drain traffic, and retire the card. Do not attempt a software workaround. Full procedure → Symptom Agent starts but reports zero accelerators; all traffic routes to fallback. Log signature device_probe: found 0 CUDA devices (CUDA_VISIBLE_DEVICES="")Root cause The container lacks the NVIDIA runtime, or `CUDA_VISIBLE_DEVICES` is empty or masked by the orchestrator. Resolution Confirm `nvidia-smi` works inside the container and that the pod requests `nvidia.com/gpu`. Full procedure → Symptom Throughput collapses at a fixed concurrency ceiling. Requests queue rather than error, until the queue itself overflows. Log signature kv_cache: no free blocks (used=100.0%), preempting seq_groupRoot cause All KV cache blocks are allocated. The engine begins preempting and recomputing sequences, which multiplies effective prefill cost. Resolution Enable FP8 KV cache quantization to roughly double block capacity, or reduce `max_num_seqs`. Full procedure → Symptom Pod restarts in a loop; readiness probe never passes. Log signature model_loader: failed to materialize weights fromRoot cause Missing or partially downloaded weights, an unreadable cache mount, or a quantization format the runtime was not built with. Resolution Verify the weight checksum and confirm the cache volume is mounted read-write with sufficient free space. Full procedure → Symptom Multi-GPU deployments hang at startup, then all ranks abort together after ~10 minutes. Log signature Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCERoot cause One rank never reached the collective — commonly a blocked NCCL port, mismatched `NCCL_SOCKET_IFNAME`, or a peer that OOMed first. Resolution Set `NCCL_DEBUG=INFO`, confirm every rank sees the same interface, and check for an upstream OOM on rank 0. Full procedure → Inference & Streaming8 Symptom RAG requests fail only when many documents are retrieved. Log signature prompt tokens 9214 + max_tokens 1024 exceeds context window 8192Root cause Retrieved context plus the reserved completion budget exceeds the model window. Retrieval count is unbounded. Resolution Enable `context.overflow_strategy` and cap retrieved chunks with a token budget rather than a document count. Full procedure → Symptom All requests fail instantly with no upstream attempt logged. Log signature router: 0/3 upstreams healthy, all circuits openRoot cause Every configured provider has tripped its circuit breaker, or health checks are misconfigured and marking healthy nodes down. Resolution Inspect breaker state, then verify the health check path returns 200 without authentication. Full procedure → Symptom Long generations are cut off at a consistent wall-clock boundary. Log signature upstream timed out after 30000ms awaiting first byteRoot cause `timeout_seconds` is shorter than the worst-case time to first token, most often because a cold fallback provider must load a model. Resolution Raise `timeout_seconds`, and prefer streaming so the connection produces bytes before the deadline. Full procedure → Symptom Sharp latency cliff at a specific request rate; p99 climbs while p50 stays flat. Log signature router: queue depth 51 exceeds max_queue_depth=50, sheddingRoot cause Arrival rate exceeds sustained service rate. The queue is doing its job — it is protecting the GPU from thrashing. Resolution Add capacity, lower `max_queue_depth` to fail fast, or configure a cloud fallback to absorb the overflow. Full procedure → Symptom SSE streams end mid-token. Browser clients see it more often than server-side clients. Log signature sse: client closed connection after 12 events (bytes_sent=2048)Root cause An intermediary — load balancer, CDN, or corporate proxy — is buffering or idle-timing the response. Resolution Disable proxy buffering, send SSE keep-alive comments, and raise the LB idle timeout above the longest generation. Full procedure → Symptom Token accounting drifts from provider billing; budget enforcement fires early or late. Log signature token_counter: tokenizer revision differs from served model revisionRoot cause The tokenizer pinned in config does not match the revision the upstream actually serves. Resolution Pin `tokenizer_revision` to the served model revision, or enable auto-detection. Full procedure → Symptom Alert fires but requests still succeed — a leading indicator, not an outage. Log signature slo: ttft p95=2841ms exceeds target 1500ms over 5m windowRoot cause Prefill is queueing behind decode work, usually from oversized batches or an absent prefix cache. Resolution Enable prompt prefix caching and chunked prefill before adding hardware. Full procedure → Symptom A burst of 429s from a cloud provider cascades into a retry storm that makes the problem worse. Log signature upstream openai returned 429; retry-after=20s; breaker half-openRoot cause Provider quota exhausted, then amplified by naive client retries without jitter. Resolution Honor `Retry-After`, enable exponential backoff with jitter, and let the circuit breaker shed load. Full procedure → Networking & Auth5 Symptom Every request from one client fails while others succeed. Log signature auth: key id=oc_live_3f2a… not found or revokedRoot cause Key revoked, rotated, or issued in a different environment than the one called. Resolution Reissue the key and confirm the client targets the matching environment. Full procedure → Symptom Authentication succeeds but specific models or routes are refused. Log signature rbac: key lacks scope models:invoke for target gpt-4oRoot cause The API key’s scope set excludes the requested model or operation. Resolution Grant the scope on the key, or route the client to a permitted model alias. Full procedure → Symptom A tenant is cut off mid-month while other tenants are unaffected. Log signature budget: tenant acme consumed 100.4% of monthly cap ($2,010/$2,000)Root cause The tenant reached its configured spend ceiling. Working as designed. Resolution Raise the cap, or configure a downgrade route to a cheaper model at the ceiling. Full procedure → Symptom Clients are throttled by Oculis itself rather than by the upstream provider. Log signature ratelimit: tenant acme exceeded 600 rpm (bucket=tenant:acme)Root cause The tenant exceeded its configured request or token rate. Resolution Raise the limit, widen the burst allowance, or move the client to its own bucket. Full procedure → Symptom Only self-hosted or on-prem upstreams fail; public providers work. Log signature x509: certificate signed by unknown authorityRoot cause A private or intercepting CA is not present in the agent trust store. Resolution Mount the CA bundle and set `tls.ca_bundle_path`. Never disable verification in production. Full procedure → Configuration3 Symptom Agent refuses to start and exits with code 78 before binding any port. Log signature config: validation failed at gateway.upstreams[1].endpoint: must be a URLRoot cause The config file failed schema validation. The path in the message is exact. Resolution Run `oculis config validate --explain` to see the failing path and expected type. Full procedure → Symptom A config that worked before an upgrade is now rejected. Log signature config: apiVersion oculis.ai/v1alpha1 is no longer supportedRoot cause The config schema version was removed in a major release. Resolution Run `oculis config migrate` to rewrite the file to the current schema version. Full procedure → Symptom Agent starts in development but fails in production with an identical config file. Log signature secrets: reference ${env:OPENAI_API_KEY} resolved to empty stringRoot cause The referenced environment variable or secret store key is absent at runtime. Resolution Confirm the secret is mounted in the deployment, then re-run `oculis status --check-deps`. Full procedure → Vector Store3 Symptom Retrieval quality degrades gradually with no configuration change. Log signature vector: collection embedded with text-embedding-3-small@v1, query uses @v2Root cause The provider silently updated a floating embedding model version. Resolution Pin the embedding model version explicitly and schedule deliberate re-embeds. Full procedure → Symptom Retrieval fails immediately after switching embedding models. Log signature vector: index dim 1536 does not match query embedding dim 3072Root cause The collection was built with a different embedding model. Dimensions are not convertible. Resolution Re-embed the corpus into a new collection, then cut over with an alias swap. Full procedure → Symptom Chat works but every answer loses its grounding citations. Log signature vector: dial tcp 10.4.2.11:6333: i/o timeoutRoot cause Network policy, DNS, or a store that is still restoring after a restart. Resolution Verify reachability from inside the pod, then enable `retrieval.fail_open` if degraded answers beat outages. Full procedure → Guardrails3 Symptom Specific prompts are refused with a policy message instead of a completion. Log signature guardrail: policy=pii_strict action=block matched=US_SSNRoot cause A guardrail policy matched. This is the intended behavior. Resolution Review the match in the audit log; if it is a false positive, narrow the detector or add an allowlist. Full procedure → Symptom Requests carrying retrieved web or document content are blocked. Log signature guardrail: prompt_injection score=0.94 threshold=0.85 source=retrieved_contextRoot cause The injection classifier flagged instruction-like text inside untrusted retrieved content. Resolution Keep the block. Fence retrieved content with `context.wrap_untrusted` so it is never read as instructions. Full procedure → Symptom Requests fail closed when the redaction service is unavailable — by design, to avoid leaking PII. Log signature guardrail: redactor unavailable, failing closed (fail_open=false)Root cause The redaction sidecar is down or unreachable and the policy is set to fail closed. Resolution Restore the redactor. Only set `fail_open: true` if unredacted traffic is acceptable for that route. Full procedure → 
