Fallback & Failover

Applies tov1.4.0Hardware2× NVIDIA A100 / H100RequiresCUDA 12.1+DifficultyIntermediateImpactRolling restart

This guide configures Oculis to route LLM requests to local GPU nodes first, and fail over to a cloud provider when the local tier saturates, degrades, or disappears entirely.

The goal is not “never fail” — it is fail predictably: shed load in a defined order, never stack retries into a stampede, and make the current state of every upstream visible.

What this changes

                     ┌──────────────────────────────────────────┐
   Client apps  ───►  │  Oculis Gateway  :8080                   │
                     │                                          │
                     │   ┌────────────┐  healthy   ┌──────────┐ │
                     │   │  Router    │───────────►│ PRIMARY  │─┼──►  vLLM  (local GPU)
                     │   │            │            │ vllm     │ │     queue depth ≤ 50
                     │   │  breaker + │  open /    └──────────┘ │
                     │   │  overflow  │  saturated ┌──────────┐ │
                     │   │            │───────────►│ FALLBACK │─┼──►  OpenAI (cloud)
                     │   └────────────┘            │ openai   │ │
                     │                             └──────────┘ │
                     └──────────────────────────────────────────┘
                                     │
                                     └──►  :9090/metrics  (breaker state, overflow rate)

Before: a single upstream, and any local failure is a client-visible outage. After: a routing decision per request, with the local tier absorbing steady state and the cloud tier absorbing bursts and outages.

Prerequisites

• Oculis Collector v1.4.0 or later — verify with oculis --version
• NVIDIA driver 535.86.05 or later, with CUDA 12.1+
• At least 2× NVIDIA A100 or H100 for a meaningful local tier
• A reachable vLLM endpoint serving an OpenAI-compatible API
• A cloud provider API key available as an environment variable or secret reference
• Network ports: 8080 inbound for the gateway listener, 9090 for metrics, plus egress 443 to the fallback provider
• Permissions: ability to restart the agent (this change requires a rolling restart)

Verify all of the aboveoculis status --check-deps

Step 1 — Define the upstreams

Declare both tiers explicitly. Ordering in upstreams does not imply priority; the routing strategy does.

CLI / YAML

oculis-config.yaml

apiVersion: oculis.ai/v1
kind: GatewayConfig

oculis:
  gateway:
    listen_port: 8080
    metrics_port: 9090

  upstreams:
    - name: vllm-primary
      provider: vllm
      endpoint: 'http://localhost:8000/v1'
      model: meta-llama/Llama-3.1-70B-Instruct
      max_queue_depth: 50 # shed above this rather than queue forever
      timeout_seconds: 30
      health_check:
        path: /health
        interval_seconds: 5
        unhealthy_threshold: 3

    - name: openai-fallback
      provider: openai
      model: gpt-4o-mini
      api_key: ${env:OPENAI_API_KEY}
      timeout_seconds: 60 # cold cloud routes are slower to first byte

Python

gateway.py

from oculis import GatewayConfig, Upstream, HealthCheck, Router

config = GatewayConfig(
    listen_port=8080,
    metrics_port=9090,
    upstreams=[
        Upstream(
            name="vllm-primary",
            provider="vllm",
            endpoint="http://localhost:8000/v1",
            model="meta-llama/Llama-3.1-70B-Instruct",
            max_queue_depth=50,
            timeout_seconds=30,
            health_check=HealthCheck(
                path="/health",
                interval_seconds=5,
                unhealthy_threshold=3,
            ),
        ),
        Upstream(
            name="openai-fallback",
            provider="openai",
            model="gpt-4o-mini",
            api_key="${env:OPENAI_API_KEY}",
            timeout_seconds=60,
        ),
    ],
)

router = Router(config)

TypeScript

gateway.ts

import { GatewayConfig, Router } from '@oculis/sdk';

const config = new GatewayConfig({
  listenPort: 8080,
  metricsPort: 9090,
  upstreams: [
    {
      name: 'vllm-primary',
      provider: 'vllm',
      endpoint: 'http://localhost:8000/v1',
      model: 'meta-llama/Llama-3.1-70B-Instruct',
      maxQueueDepth: 50,
      timeoutSeconds: 30,
      healthCheck: { path: '/health', intervalSeconds: 5, unhealthyThreshold: 3 },
    },
    {
      name: 'openai-fallback',
      provider: 'openai',
      model: 'gpt-4o-mini',
      apiKey: '${env:OPENAI_API_KEY}',
      timeoutSeconds: 60,
    },
  ],
});

export const router = new Router(config);

Helm

values.yaml

gateway:
  listenPort: 8080
  metricsPort: 9090

upstreams:
  - name: vllm-primary
    provider: vllm
    endpoint: http://vllm.inference.svc.cluster.local:8000/v1
    model: meta-llama/Llama-3.1-70B-Instruct
    maxQueueDepth: 50
    timeoutSeconds: 30
    healthCheck:
      path: /health
      intervalSeconds: 5
      unhealthyThreshold: 3

  - name: openai-fallback
    provider: openai
    model: gpt-4o-mini
    apiKeySecretRef:
      name: oculis-provider-keys
      key: openai
    timeoutSeconds: 60

Parameters that matter most:

Parameter	Why it matters
max_queue_depth	The saturation point. Too high and requests time out while queued; too low and you spill to the expensive tier prematurely. Start at ~2× steady-state depth.
timeout_seconds	Must exceed worst-case time to first token. A cold fallback model can take 10s+ before its first byte.
unhealthy_threshold	Consecutive failed checks before eviction. Below 3, a single GC pause evicts a healthy node.
api_key	Use a ${env:…} or secret reference — never a literal. An unresolved reference produces ERR_OCULIS_SECRET_UNRESOLVED at startup.

Step 2 — Choose a routing strategy

CLI / YAML

oculis-config.yaml

oculis:
  routes:
    - name: default
      strategy: primary_with_fallback
      primary: vllm-primary
      fallback:
        - openai-fallback
      overflow:
        on_queue_saturated: fallback # spill instead of shedding
        on_upstream_error: fallback
        on_timeout: fallback
      retry:
        max_attempts: 2
        backoff: exponential
        initial_delay_ms: 200
        jitter: full # required — see the warning below

Python

gateway.py

from oculis import Route, Overflow, Retry

route = Route(
    name="default",
    strategy="primary_with_fallback",
    primary="vllm-primary",
    fallback=["openai-fallback"],
    overflow=Overflow(
        on_queue_saturated="fallback",
        on_upstream_error="fallback",
        on_timeout="fallback",
    ),
    retry=Retry(
        max_attempts=2,
        backoff="exponential",
        initial_delay_ms=200,
        jitter="full",
    ),
)

config.routes = [route]

cURL

Apply at runtime via the admin API

curl -X PUT http://localhost:8080/admin/v1/routes/default \
  -H "Authorization: Bearer $OCULIS_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "strategy": "primary_with_fallback",
    "primary": "vllm-primary",
    "fallback": ["openai-fallback"],
    "overflow": {
      "on_queue_saturated": "fallback",
      "on_upstream_error": "fallback",
      "on_timeout": "fallback"
    },
    "retry": { "max_attempts": 2, "backoff": "exponential", "jitter": "full" }
  }'

Available strategies

Strategy	Behavior	Use when
primary_with_fallback	All traffic to primary; spill on saturation, error, or timeout.	You own GPUs and want cloud only as insurance.
round_robin	Even distribution across healthy upstreams.	Homogeneous nodes behind one model.
least_queue	Route to the upstream with the shallowest queue.	Heterogeneous node sizes, or uneven request costs.
weighted	Fixed percentage split.	Canarying a new node or model version.
semantic	Classify the prompt, route cheap requests to a small model.	Cost reduction — see throughput.

Always set jitter

Retries without jitter synchronize. When a shared upstream returns a burst of 429s, every client backs off by the same interval and retries in the same millisecond, which reproduces the overload exactly. jitter: full is the only safe setting under load, and it is why ERR_OCULIS_UPSTREAM_RATE_LIMIT cascades in the first place.

Step 3 — Configure the circuit breaker

The breaker stops Oculis from spending its timeout budget on an upstream that is already known to be down.

CLI / YAML

oculis-config.yaml

oculis:
  upstreams:
    - name: vllm-primary
      circuit_breaker:
        failure_threshold: 5 # consecutive failures before opening
        open_duration_seconds: 30 # reject fast for this long
        half_open_probes: 1 # trial requests before closing again

Python

gateway.py

from oculis import CircuitBreaker

config.upstreams[0].circuit_breaker = CircuitBreaker(
    failure_threshold=5,
    open_duration_seconds=30,
    half_open_probes=1,
)

Circuit breaker behavior

The breaker is a three-state machine, and every state is visible in metrics:

State	What happens	Metric value
Closed	Traffic flows normally. Failures increment a counter.	oculis_breaker_state{state="closed"}
Open	Requests fail immediately and route to fallback. No upstream call.	oculis_breaker_state{state="open"}
Half-open	A limited number of probes test recovery.	oculis_breaker_state{state="half_open"}

If every upstream breaker is open, requests fail with ERR_OCULIS_NO_HEALTHY_UPSTREAM — the gateway is telling you it has nowhere left to send traffic.

Overflow routing

on_queue_saturated: fallback is what converts ERR_OCULIS_QUEUE_SATURATED from a client-visible 503 into a silent, more expensive success. That trade-off should be deliberate: it protects availability and spends money. Set a spend budget alongside it so a sustained local outage cannot produce a surprise invoice.

Step 4 — Apply and roll out

1. Validate the config before it touches a running process.

   oculis config validate --file oculis-config.yaml --explain

2. Apply with a rolling restart so in-flight requests drain rather than drop.

   oculis apply --file oculis-config.yaml --strategy rolling --drain-timeout 60s

3. Watch the rollout until every replica reports the new config generation.

   oculis status --watch

Verification

These three checks prove the primary works, the fallback works, and the breaker recovers. Run all three — a fallback that has never been exercised is not a fallback.

1. Confirm both upstreams are reachable and the route resolves.

oculis gateway test --route default

Expected output

[INFO]  Resolving route "default" (strategy=primary_with_fallback)
[INFO]  Primary   vllm-primary   http://localhost:8000/v1   OK   (latency 12ms)
[INFO]  Fallback  openai-fallback  api.openai.com           OK   (latency 148ms)
[PASS]  Route "default" resolved successfully.

2. Force a primary failure and confirm traffic actually lands on the fallback.

oculis gateway test --route default --simulate-failure primary

Expected output

[INFO]  Primary endpoint check (vllm:8000)... OK (latency 12ms)
[WARN]  Simulating primary failure (injected: connection refused)
[INFO]  Breaker vllm-primary: closed -> open (failures 5/5)
[INFO]  Routing request to fallback (openai-fallback)... SUCCESS (HTTP 200 OK)
[INFO]  Breaker vllm-primary: open -> half_open after 30s
[INFO]  Probe succeeded; breaker vllm-primary: half_open -> closed
[PASS]  Fallback circuit breaker verified successfully.

3. Confirm the metrics an alert would fire on are actually being exported.

curl -s http://localhost:9090/metrics | grep -E 'oculis_(breaker_state|overflow_total)'

Expected output

oculis_breaker_state{upstream="vllm-primary",state="closed"} 1
oculis_breaker_state{upstream="openai-fallback",state="closed"} 1
oculis_overflow_total{route="default",reason="queue_saturated"} 0
oculis_overflow_total{route="default",reason="upstream_error"} 1

The 1 on upstream_error is the simulated failure from check 2. If it reads 0, overflow is not wired up and your fallback will not engage under real load.

Edge cases and known limitations

Model behavior differs across the failover boundary

Your primary and fallback are different models. Prompts tuned for a 70B local model may return measurably worse output from gpt-4o-mini, and structured-output schemas may not be honored identically. Test your prompts against both upstreams, and treat fallback responses as a degraded mode you have consciously accepted.

• Streaming responses cannot fail over mid-stream. Once the first token is sent, the response is committed to that upstream. A failure after that point surfaces to the client as ERR_OCULIS_STREAM_ABORTED. Failover only applies before the first byte.
• Token accounting differs by provider. Budget enforcement uses each provider’s own tokenizer. If they disagree, you will see ERR_OCULIS_TOKENIZER_MISMATCH — pin tokenizer_revision per upstream.
• Do not enable both on_timeout: fallback and a long primary timeout. A 120s primary timeout means clients wait two minutes before failover even begins. Keep the primary timeout tight and let the fallback absorb.
• Health checks must not require authentication. A /health path behind auth will return 401, the node will be marked unhealthy, and all traffic will sit on the fallback while the primary is perfectly fine.
• max_queue_depth is per replica, not per cluster. Three replicas at depth 50 tolerate 150 queued requests in aggregate.

Failure modes this configuration produces

Error code	Symptom & log signature	Root cause & first action
ErrorERR_OCULIS_QUEUE_SATURATEDHTTP 503	Sharp latency cliff at a specific request rate; p99 climbs while p50 stays flat. router: queue depth 51 exceeds max_queue_depth=50, shedding	Arrival rate exceeds sustained service rate. The queue is doing its job — it is protecting the GPU from thrashing. FixAdd capacity, lower `max_queue_depth` to fail fast, or configure a cloud fallback to absorb the overflow. Full procedure
ErrorERR_OCULIS_PROXY_TIMEOUTHTTP 504	Long generations are cut off at a consistent wall-clock boundary. upstream timed out after 30000ms awaiting first byte	`timeout_seconds` is shorter than the worst-case time to first token, most often because a cold fallback provider must load a model. FixRaise `timeout_seconds`, and prefer streaming so the connection produces bytes before the deadline. Full procedure
WarningERR_OCULIS_STREAM_ABORTEDHTTP 499	SSE streams end mid-token. Browser clients see it more often than server-side clients. sse: client closed connection after 12 events (bytes_sent=2048)	An intermediary — load balancer, CDN, or corporate proxy — is buffering or idle-timing the response. FixDisable proxy buffering, send SSE keep-alive comments, and raise the LB idle timeout above the longest generation. Full procedure
ErrorERR_OCULIS_CONTEXT_OVERFLOWHTTP 400	RAG requests fail only when many documents are retrieved. prompt tokens 9214 + max_tokens 1024 exceeds context window 8192	Retrieved context plus the reserved completion budget exceeds the model window. Retrieval count is unbounded. FixEnable `context.overflow_strategy` and cap retrieved chunks with a token budget rather than a document count. Full procedure
ErrorERR_OCULIS_UPSTREAM_RATE_LIMITHTTP 429	A burst of 429s from a cloud provider cascades into a retry storm that makes the problem worse. upstream openai returned 429; retry-after=20s; breaker half-open	Provider quota exhausted, then amplified by naive client retries without jitter. FixHonor `Retry-After`, enable exponential backoff with jitter, and let the circuit breaker shed load. Full procedure
CriticalERR_OCULIS_NO_HEALTHY_UPSTREAMHTTP 503	All requests fail instantly with no upstream attempt logged. router: 0/3 upstreams healthy, all circuits open	Every configured provider has tripped its circuit breaker, or health checks are misconfigured and marking healthy nodes down. FixInspect breaker state, then verify the health check path returns 200 without authentication. Full procedure
WarningERR_OCULIS_TOKENIZER_MISMATCHHTTP 500	Token accounting drifts from provider billing; budget enforcement fires early or late. token_counter: tokenizer revision differs from served model revision	The tokenizer pinned in config does not match the revision the upstream actually serves. FixPin `tokenizer_revision` to the served model revision, or enable auto-detection. Full procedure
WarningERR_OCULIS_TTFT_SLO_BREACH	Alert fires but requests still succeed — a leading indicator, not an outage. slo: ttft p95=2841ms exceeds target 1500ms over 5m window	Prefill is queueing behind decode work, usually from oversized batches or an absent prefix cache. FixEnable prompt prefix caching and chunked prefill before adding hardware. Full procedure

Next steps

Rate limits & token budgetsCap what a fallback outage can cost you before it becomes an invoice.

Telemetry & observabilityAlert on breaker state and overflow rate rather than discovering it later.

Throughput & costSemantic routing sends cheap requests to small models before fallback is needed.