Inference & Streaming

Applies tov1.4.0DifficultyIntermediate

These failures share a property that makes them frustrating: the model is usually fine. The problem is almost always in queueing, buffering, timeouts, or accounting — the plumbing around inference rather than inference itself.

The per-hop trace is the fastest way to prove where time actually went:

oculis trace get <request-id> --format tree

Error code	Symptom & log signature	Root cause & first action
ErrorERR_OCULIS_QUEUE_SATURATEDHTTP 503	Sharp latency cliff at a specific request rate; p99 climbs while p50 stays flat. router: queue depth 51 exceeds max_queue_depth=50, shedding	Arrival rate exceeds sustained service rate. The queue is doing its job — it is protecting the GPU from thrashing. FixAdd capacity, lower `max_queue_depth` to fail fast, or configure a cloud fallback to absorb the overflow. Full procedure
ErrorERR_OCULIS_PROXY_TIMEOUTHTTP 504	Long generations are cut off at a consistent wall-clock boundary. upstream timed out after 30000ms awaiting first byte	`timeout_seconds` is shorter than the worst-case time to first token, most often because a cold fallback provider must load a model. FixRaise `timeout_seconds`, and prefer streaming so the connection produces bytes before the deadline. Full procedure
WarningERR_OCULIS_STREAM_ABORTEDHTTP 499	SSE streams end mid-token. Browser clients see it more often than server-side clients. sse: client closed connection after 12 events (bytes_sent=2048)	An intermediary — load balancer, CDN, or corporate proxy — is buffering or idle-timing the response. FixDisable proxy buffering, send SSE keep-alive comments, and raise the LB idle timeout above the longest generation. Full procedure
ErrorERR_OCULIS_CONTEXT_OVERFLOWHTTP 400	RAG requests fail only when many documents are retrieved. prompt tokens 9214 + max_tokens 1024 exceeds context window 8192	Retrieved context plus the reserved completion budget exceeds the model window. Retrieval count is unbounded. FixEnable `context.overflow_strategy` and cap retrieved chunks with a token budget rather than a document count. Full procedure
ErrorERR_OCULIS_UPSTREAM_RATE_LIMITHTTP 429	A burst of 429s from a cloud provider cascades into a retry storm that makes the problem worse. upstream openai returned 429; retry-after=20s; breaker half-open	Provider quota exhausted, then amplified by naive client retries without jitter. FixHonor `Retry-After`, enable exponential backoff with jitter, and let the circuit breaker shed load. Full procedure
CriticalERR_OCULIS_NO_HEALTHY_UPSTREAMHTTP 503	All requests fail instantly with no upstream attempt logged. router: 0/3 upstreams healthy, all circuits open	Every configured provider has tripped its circuit breaker, or health checks are misconfigured and marking healthy nodes down. FixInspect breaker state, then verify the health check path returns 200 without authentication. Full procedure
WarningERR_OCULIS_TOKENIZER_MISMATCHHTTP 500	Token accounting drifts from provider billing; budget enforcement fires early or late. token_counter: tokenizer revision differs from served model revision	The tokenizer pinned in config does not match the revision the upstream actually serves. FixPin `tokenizer_revision` to the served model revision, or enable auto-detection. Full procedure
WarningERR_OCULIS_TTFT_SLO_BREACH	Alert fires but requests still succeed — a leading indicator, not an outage. slo: ttft p95=2841ms exceeds target 1500ms over 5m window	Prefill is queueing behind decode work, usually from oversized batches or an absent prefix cache. FixEnable prompt prefix caching and chunked prefill before adding hardware. Full procedure

Gateway timeouts

Signature: ERR_OCULIS_PROXY_TIMEOUT

upstream timed out after 30000ms awaiting first byte

Long generations cut off at a consistent wall-clock boundary. The consistency is the clue — a real model failure is not this punctual.

Find which timeout fired

There are usually four, and the shortest wins:

Timeout	Where	Typical default
gateway.request_timeout_seconds	Oculis, global	120 s
upstreams[].timeout_seconds	Oculis, per upstream	30 s
Load balancer idle timeout	Infrastructure	60 s
Client HTTP timeout	Your application	30–60 s

oculis config show --resolved | grep -E 'timeout'

1. Confirm it is time-to-first-byte, not total duration. Check the trace: if oculis.queue.wait dominates, the upstream never started work and the fix is capacity, not a longer timeout.

2. Raise the timeout only where the wait is legitimate. A cold cloud fallback loading a model genuinely needs 60 s.

   upstreams:
     - name: openai-fallback
       timeout_seconds: 60

3. Prefer streaming. A streaming response produces bytes almost immediately, so time-to-first-byte timeouts stop firing even when total generation is long.

Raising a timeout is usually treating the symptom

A timeout that fires under load is reporting saturation. Raising it converts fast failures into slow ones and increases the number of requests waiting on a sick upstream. Check queue depth before you change a number.

Streams drop mid-response

Signature: ERR_OCULIS_STREAM_ABORTED

sse: client closed connection after 12 events (bytes_sent=2048)

Note bytes_sent=2048. That number is the tell: it is a buffer size, which means something between Oculis and the client is buffering the stream rather than passing it through.

Fix the intermediary

nginx

location /v1/ {
    proxy_pass http://oculis:8080;
    proxy_http_version 1.1;

    proxy_buffering off;          # the critical one
    proxy_cache off;
    proxy_set_header Connection '';
    chunked_transfer_encoding on;

    proxy_read_timeout 600s;
    proxy_send_timeout 600s;
}

Envoy / Istio

route:
  timeout: 0s # disable the route timeout for streaming
  idleTimeout: 600s
typed_per_filter_config:
  envoy.filters.http.buffer:
    '@type': type.googleapis.com/envoy.extensions.filters.http.buffer.v3.BufferPerRoute
    disabled: true

AWS ALB

aws elbv2 modify-load-balancer-attributes \
  --load-balancer-arn "$ALB_ARN" \
  --attributes Key=idle_timeout.timeout_seconds,Value=600

ALB does not buffer, but its 60 s default idle timeout will cut streams with slow first tokens.

Cloudflare

Cloudflare buffers by default on some plans. Add a Configuration Rule disabling buffering for the API path, or route /v1/* through a subdomain with proxying disabled (grey cloud).

Then enable keep-alives so idle timers never see an idle connection:

oculis-config.yaml

oculis:
  gateway:
    keepalive_seconds: 75 # must exceed the LB idle timeout
    sse:
      heartbeat_seconds: 15 # emits `: keepalive` comments between tokens

Streaming responses cannot fail over

Once the first token is sent, the response is committed to that upstream. A failure after that point cannot be retried on a fallback — the client has already received partial output. Failover only applies before the first byte. See fallback edge cases.

Rate limit cascades

Signature: ERR_OCULIS_UPSTREAM_RATE_LIMIT

upstream openai returned 429; retry-after=20s; breaker half-open

A burst of 429s triggers retries; the retries arrive together and reproduce the overload. The graph looks like a sawtooth that never recovers.

Why it self-amplifies

Without jitter, every client backs off by the same interval and retries in the same millisecond. The retry storm is more synchronized than the original traffic.

oculis-config.yaml

routes:
  - name: default
    retry:
      max_attempts: 2 # not 5 — retries multiply load
      jitter: full # mandatory under load
      backoff: exponential
      initial_delay_ms: 200
      respect_retry_after: true # honor the provider's own guidance

1. Set jitter: full. This is the single highest-impact change.
2. Cap max_attempts at 2. Each additional attempt multiplies load on an upstream that is already telling you to stop.
3. Honor Retry-After. The provider knows when it will be ready; guessing is worse.
4. Let the breaker shed. An open circuit fails fast and gives the upstream room to recover — see circuit breaker behavior.
5. Add a fallback so shed traffic has somewhere to go rather than failing.

Context window overflow

Signature: ERR_OCULIS_CONTEXT_OVERFLOW

prompt tokens 9214 + max_tokens 1024 exceeds context window 8192

Note that the check includes max_tokens. The prompt alone fits; the prompt plus the reserved completion budget does not. This is why raising max_tokens can break requests that previously worked.

Almost always a RAG problem: retrieval is bounded by document count rather than tokens.

oculis-config.yaml

oculis:
  retrieval:
    - name: docs-retrieval
      top_k: 8
      max_context_tokens: 3000 # bound by tokens, not documents
      overflow_strategy: truncate_lowest_score

  context:
    reserve_completion_tokens: 1024 # subtract before assembling
    overflow_strategy: truncate_lowest_score

Confirm the budget arithmetic for a real request:

oculis context inspect --request-id req_01HQ8Z3K4M5N6P7Q8R9S

Example output

  model window            8,192
  system prompt             182
  retrieved context       6,914   ← unbounded
  user message              118
  reserved completion     1,024
  ---------------------  ------
  total                   8,238   OVER by 46

Token accounting drift

Signature: ERR_OCULIS_TOKENIZER_MISMATCH

token_counter: tokenizer revision differs from served model revision

Budgets fire early or late, and your cost report disagrees with the provider invoice.

upstreams:
  - name: vllm-primary
    tokenizer_revision: 'main@a1b2c3d' # pin to the served revision
    # or:
    tokenizer_autodetect: true # query the upstream at startup

oculis tokens verify --upstream vllm-primary --sample 100

Expected output

  sampled            100 requests
  oculis count       412,884 tokens
  upstream count     412,884 tokens
  drift              0.00%
[PASS] Token accounting matches upstream within 0.5% tolerance.

Drift above ~1% will make spend budgets meaningfully wrong over a month.

High time to first token

TTFT is what users perceive as speed, and it is usually a queueing problem rather than a model problem. Confirm which before changing anything:

oculis trace get <request-id> --format tree | grep -E 'queue.wait|prefill|decode'

Example output

├── oculis.queue.wait     depth=12    412ms   ← queueing dominates
└── oculis.upstream                 1,290ms
    ├── oculis.prefill    tokens=3891 380ms
    └── oculis.decode     tokens=512  910ms

Dominant span	Meaning	Go to
oculis.queue.wait	Saturation. Not a model problem.	capacity planning
oculis.prefill	Long prompts, no prefix cache.	reducing TTFT
oculis.retrieval	Slow vector store.	vector databases
oculis.guardrail	Response-direction buffering.	guardrails

Sustained breaches emit ERR_OCULIS_TTFT_SLO_BREACH as a leading indicator — requests still succeed at that point.

Verification

Confirms streaming survives the full path including your proxy, and that timeouts and retries behave under induced failure. Test through the load balancer, not against the gateway directly — the proxy is usually the culprit.

1. Streaming works end to end, through your real ingress.

curl -N -s https://gateway.example.com/v1/chat/completions \
  -H "Authorization: Bearer $OCULIS_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"default","stream":true,
       "messages":[{"role":"user","content":"Count slowly from 1 to 50."}]}' \
  | head -20

Expected output

data: {"choices":[{"delta":{"content":"1"}}]}
data: {"choices":[{"delta":{"content":", 2"}}]}
: keepalive
data: {"choices":[{"delta":{"content":", 3"}}]}

Tokens should appear incrementally. If the whole response arrives at once after a pause, something in the path is buffering.

2. Retry behavior is sane under induced 429s.

oculis gateway test --route default --simulate-rate-limit

Expected output

[WARN]  Injecting 429 from upstream openai (retry-after=2s)
[INFO]  Attempt 1 failed; backing off 217ms (jitter=full)
[INFO]  Attempt 2 honored Retry-After=2s
[INFO]  Breaker openai: closed -> half_open
[PASS]  Retry policy respects Retry-After and applies jitter.

If the log shows identical backoff intervals across attempts, jitter is not configured and you will see cascades under real load.

Next steps

Latency tuningPrefix caching, chunked prefill, and KV cache quantization.

Fallback & failoverCircuit breakers and overflow routing to absorb saturation.

Networking & authIf the failure is before the model rather than during it.