Diagnostic Matrix
You arrived here holding something specific: a red line in a log, a status code, or a graph that moved. This page is organized around that, not around topics.
Search the matrix below by error code (ERR_OCULIS_CUDA_OOM), by HTTP status (504), by a
fragment of a log line (no free blocks), or by what you observe (stream drops). Every row links
to the full procedure.
| Error code | Symptom & log signature | Root cause & first action |
|---|---|---|
CriticalERR_OCULIS_CUDA_OOMHTTP 503 | Requests fail intermittently under load; failures cluster on long prompts rather than long generations. torch.OutOfMemoryError: CUDA out of memory. Tried to allocate | Peak prefill activation memory plus the KV cache exceeds free VRAM. Usually caused by `max_model_len` or `max_num_batched_tokens` set higher than the card can hold. FixLower `gpu_memory_utilization` to 0.88, cap `max_num_batched_tokens`, then size the KV cache with the capacity planner. Full procedure |
ErrorERR_OCULIS_KV_CACHE_EXHAUSTEDHTTP 503 | Throughput collapses at a fixed concurrency ceiling. Requests queue rather than error, until the queue itself overflows. kv_cache: no free blocks (used=100.0%), preempting seq_group | All KV cache blocks are allocated. The engine begins preempting and recomputing sequences, which multiplies effective prefill cost. FixEnable FP8 KV cache quantization to roughly double block capacity, or reduce `max_num_seqs`. Full procedure |
CriticalERR_OCULIS_GPU_UNAVAILABLEHTTP 503 | Agent starts but reports zero accelerators; all traffic routes to fallback. device_probe: found 0 CUDA devices (CUDA_VISIBLE_DEVICES="") | The container lacks the NVIDIA runtime, or `CUDA_VISIBLE_DEVICES` is empty or masked by the orchestrator. FixConfirm `nvidia-smi` works inside the container and that the pod requests `nvidia.com/gpu`. Full procedure |
CriticalERR_OCULIS_DRIVER_MISMATCHHTTP 500 | Agent exits during model load, immediately after the CUDA context is created. CUDA error: forward compatibility was attempted on non supported HW | The CUDA runtime bundled in the image is newer than the host kernel driver supports. FixMatch the driver to the runtime using the compatibility matrix, or install the forward-compat package on the host. Full procedure |
CriticalERR_OCULIS_NCCL_TIMEOUTHTTP 500 | Multi-GPU deployments hang at startup, then all ranks abort together after ~10 minutes. Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE | One rank never reached the collective — commonly a blocked NCCL port, mismatched `NCCL_SOCKET_IFNAME`, or a peer that OOMed first. FixSet `NCCL_DEBUG=INFO`, confirm every rank sees the same interface, and check for an upstream OOM on rank 0. Full procedure |
CriticalERR_OCULIS_ECC_UNCORRECTABLEHTTP 500 | A single node produces corrupted output or crashes repeatedly; others are healthy. Xid 48: Double Bit ECC Error | Failing GPU memory. This is a hardware fault, not a configuration problem. FixCordon the node immediately, drain traffic, and retire the card. Do not attempt a software workaround. Full procedure |
ErrorERR_OCULIS_MODEL_LOAD_FAILEDHTTP 500 | Pod restarts in a loop; readiness probe never passes. model_loader: failed to materialize weights from | Missing or partially downloaded weights, an unreadable cache mount, or a quantization format the runtime was not built with. FixVerify the weight checksum and confirm the cache volume is mounted read-write with sufficient free space. Full procedure |
ErrorERR_OCULIS_QUEUE_SATURATEDHTTP 503 | Sharp latency cliff at a specific request rate; p99 climbs while p50 stays flat. router: queue depth 51 exceeds max_queue_depth=50, shedding | Arrival rate exceeds sustained service rate. The queue is doing its job — it is protecting the GPU from thrashing. FixAdd capacity, lower `max_queue_depth` to fail fast, or configure a cloud fallback to absorb the overflow. Full procedure |
ErrorERR_OCULIS_PROXY_TIMEOUTHTTP 504 | Long generations are cut off at a consistent wall-clock boundary. upstream timed out after 30000ms awaiting first byte | `timeout_seconds` is shorter than the worst-case time to first token, most often because a cold fallback provider must load a model. FixRaise `timeout_seconds`, and prefer streaming so the connection produces bytes before the deadline. Full procedure |
WarningERR_OCULIS_STREAM_ABORTEDHTTP 499 | SSE streams end mid-token. Browser clients see it more often than server-side clients. sse: client closed connection after 12 events (bytes_sent=2048) | An intermediary — load balancer, CDN, or corporate proxy — is buffering or idle-timing the response. FixDisable proxy buffering, send SSE keep-alive comments, and raise the LB idle timeout above the longest generation. Full procedure |
ErrorERR_OCULIS_CONTEXT_OVERFLOWHTTP 400 | RAG requests fail only when many documents are retrieved. prompt tokens 9214 + max_tokens 1024 exceeds context window 8192 | Retrieved context plus the reserved completion budget exceeds the model window. Retrieval count is unbounded. FixEnable `context.overflow_strategy` and cap retrieved chunks with a token budget rather than a document count. Full procedure |
ErrorERR_OCULIS_UPSTREAM_RATE_LIMITHTTP 429 | A burst of 429s from a cloud provider cascades into a retry storm that makes the problem worse. upstream openai returned 429; retry-after=20s; breaker half-open | Provider quota exhausted, then amplified by naive client retries without jitter. FixHonor `Retry-After`, enable exponential backoff with jitter, and let the circuit breaker shed load. Full procedure |
CriticalERR_OCULIS_NO_HEALTHY_UPSTREAMHTTP 503 | All requests fail instantly with no upstream attempt logged. router: 0/3 upstreams healthy, all circuits open | Every configured provider has tripped its circuit breaker, or health checks are misconfigured and marking healthy nodes down. FixInspect breaker state, then verify the health check path returns 200 without authentication. Full procedure |
WarningERR_OCULIS_TOKENIZER_MISMATCHHTTP 500 | Token accounting drifts from provider billing; budget enforcement fires early or late. token_counter: tokenizer revision differs from served model revision | The tokenizer pinned in config does not match the revision the upstream actually serves. FixPin `tokenizer_revision` to the served model revision, or enable auto-detection. Full procedure |
WarningERR_OCULIS_TTFT_SLO_BREACH | Alert fires but requests still succeed — a leading indicator, not an outage. slo: ttft p95=2841ms exceeds target 1500ms over 5m window | Prefill is queueing behind decode work, usually from oversized batches or an absent prefix cache. FixEnable prompt prefix caching and chunked prefill before adding hardware. Full procedure |
ErrorERR_OCULIS_UPSTREAM_TLS_VERIFYHTTP 502 | Only self-hosted or on-prem upstreams fail; public providers work. x509: certificate signed by unknown authority | A private or intercepting CA is not present in the agent trust store. FixMount the CA bundle and set `tls.ca_bundle_path`. Never disable verification in production. Full procedure |
ErrorERR_OCULIS_AUTH_INVALID_KEYHTTP 401 | Every request from one client fails while others succeed. auth: key id=oc_live_3f2a… not found or revoked | Key revoked, rotated, or issued in a different environment than the one called. FixReissue the key and confirm the client targets the matching environment. Full procedure |
ErrorERR_OCULIS_AUTH_SCOPE_DENIEDHTTP 403 | Authentication succeeds but specific models or routes are refused. rbac: key lacks scope models:invoke for target gpt-4o | The API key’s scope set excludes the requested model or operation. FixGrant the scope on the key, or route the client to a permitted model alias. Full procedure |
WarningERR_OCULIS_BUDGET_EXCEEDEDHTTP 402 | A tenant is cut off mid-month while other tenants are unaffected. budget: tenant acme consumed 100.4% of monthly cap ($2,010/$2,000) | The tenant reached its configured spend ceiling. Working as designed. FixRaise the cap, or configure a downgrade route to a cheaper model at the ceiling. Full procedure |
WarningERR_OCULIS_RATE_LIMIT_LOCALHTTP 429 | Clients are throttled by Oculis itself rather than by the upstream provider. ratelimit: tenant acme exceeded 600 rpm (bucket=tenant:acme) | The tenant exceeded its configured request or token rate. FixRaise the limit, widen the burst allowance, or move the client to its own bucket. Full procedure |
CriticalERR_OCULIS_CONFIG_INVALID | Agent refuses to start and exits with code 78 before binding any port. config: validation failed at gateway.upstreams[1].endpoint: must be a URL | The config file failed schema validation. The path in the message is exact. FixRun `oculis config validate --explain` to see the failing path and expected type. Full procedure |
ErrorERR_OCULIS_CONFIG_VERSION_UNSUPPORTED | A config that worked before an upgrade is now rejected. config: apiVersion oculis.ai/v1alpha1 is no longer supported | The config schema version was removed in a major release. FixRun `oculis config migrate` to rewrite the file to the current schema version. Full procedure |
CriticalERR_OCULIS_SECRET_UNRESOLVED | Agent starts in development but fails in production with an identical config file. secrets: reference ${env:OPENAI_API_KEY} resolved to empty string | The referenced environment variable or secret store key is absent at runtime. FixConfirm the secret is mounted in the deployment, then re-run `oculis status --check-deps`. Full procedure |
ErrorERR_OCULIS_VECTOR_DIM_MISMATCHHTTP 400 | Retrieval fails immediately after switching embedding models. vector: index dim 1536 does not match query embedding dim 3072 | The collection was built with a different embedding model. Dimensions are not convertible. FixRe-embed the corpus into a new collection, then cut over with an alias swap. Full procedure |
ErrorERR_OCULIS_VECTOR_STORE_UNREACHABLEHTTP 502 | Chat works but every answer loses its grounding citations. vector: dial tcp 10.4.2.11:6333: i/o timeout | Network policy, DNS, or a store that is still restoring after a restart. FixVerify reachability from inside the pod, then enable `retrieval.fail_open` if degraded answers beat outages. Full procedure |
WarningERR_OCULIS_EMBEDDING_MODEL_DRIFTHTTP 409 | Retrieval quality degrades gradually with no configuration change. vector: collection embedded with text-embedding-3-small@v1, query uses @v2 | The provider silently updated a floating embedding model version. FixPin the embedding model version explicitly and schedule deliberate re-embeds. Full procedure |
WarningERR_OCULIS_GUARDRAIL_BLOCKEDHTTP 451 | Specific prompts are refused with a policy message instead of a completion. guardrail: policy=pii_strict action=block matched=US_SSN | A guardrail policy matched. This is the intended behavior. FixReview the match in the audit log; if it is a false positive, narrow the detector or add an allowlist. Full procedure |
WarningERR_OCULIS_INJECTION_DETECTEDHTTP 451 | Requests carrying retrieved web or document content are blocked. guardrail: prompt_injection score=0.94 threshold=0.85 source=retrieved_context | The injection classifier flagged instruction-like text inside untrusted retrieved content. FixKeep the block. Fence retrieved content with `context.wrap_untrusted` so it is never read as instructions. Full procedure |
CriticalERR_OCULIS_PII_REDACTION_FAILEDHTTP 500 | Requests fail closed when the redaction service is unavailable — by design, to avoid leaking PII. guardrail: redactor unavailable, failing closed (fail_open=false) | The redaction sidecar is down or unreachable and the policy is set to fail closed. FixRestore the redactor. Only set `fail_open: true` if unredacted traffic is acceptable for that route. Full procedure |
No entry matches that filter. Try the error code alone, or search the whole site with ⌘K.
Showing 29 of 29 documented failure modes.
How to read a severity
Section titled “How to read a severity”The severity describes what the failure does to traffic, not how hard it is to fix.
| Severity | Meaning | Expected response |
|---|---|---|
| Critical | Traffic is failing or the agent will not start. No workaround inside Oculis. | Page an operator. Fix or fail over now. |
| Error | A subset of requests fail. Fallback or shedding is absorbing the rest. | Fix within the current on-call shift. |
| Warning | Requests still succeed. This is a leading indicator or a policy decision. | Investigate before it becomes an Error. |
When nothing in the matrix matches
Section titled “When nothing in the matrix matches”Work the problem in this order. It moves from cheapest to most expensive, and each step narrows what the next one has to consider.
-
Confirm the agent’s own view of the world.
Terminal window oculis status --check-deps --verboseThis validates config, resolves every secret reference, probes each upstream, and reports GPU visibility. Most “mysterious” failures are a config or secret that never resolved.
-
Establish which side of the gateway is failing.
Terminal window oculis gateway test --route default --debugIf this succeeds, Oculis is healthy and the problem is between your client and the gateway. If it fails, the problem is between the gateway and an upstream.
-
Look at the request, not the aggregate.
Find one failing request ID in your logs and pull its full trace:
Terminal window oculis trace get <request-id> --format treeThe tree shows time spent in each hop — guardrails, retrieval, queueing, prefill, decode — so you can see where latency or failure actually originated.
-
Widen the log level for one route only.
Raising the global log level on a busy gateway buries the signal. Scope it:
Terminal window oculis config set observability.log_overrides.route:default=debug --reloadRemember to remove the override afterwards.
-
Collect a support bundle and escalate. See collecting diagnostics.