Gateway & Routing

Applies tov1.4.0HardwareAny (GPU optional)DifficultyBeginnerImpactRolling restart

The gateway is the listener your applications talk to, and the router is what decides where each request actually goes. This page covers both, plus health checking and model aliasing.

If you want resilient multi-provider failover specifically, read fallback & failover after this page.

Prerequisites

• Oculis Collector v1.4.0 or later
• At least one reachable model endpoint — a local runtime or a cloud provider API key
• Ports 8080 inbound for the gateway and 9090 inbound for metrics
• Permissions: ability to restart the agent, or an admin API key for runtime changes

Verify all of the aboveoculis status --check-deps

Step 1 — Configure the listener

CLI / YAML

oculis-config.yaml

apiVersion: oculis.ai/v1
kind: GatewayConfig

oculis:
  gateway:
    listen_address: '0.0.0.0'
    listen_port: 8080
    metrics_port: 9090
    max_request_bytes: 8_388_608 # 8 MiB
    request_timeout_seconds: 120 # hard ceiling for any single request
    keepalive_seconds: 75 # must exceed your load balancer idle timeout
    cors:
      allowed_origins: ['https://app.example.com']
      allow_credentials: true

Helm

values.yaml

gateway:
  listenAddress: '0.0.0.0'
  listenPort: 8080
  metricsPort: 9090
  maxRequestBytes: 8388608
  requestTimeoutSeconds: 120
  keepaliveSeconds: 75
  cors:
    allowedOrigins:
      - https://app.example.com
    allowCredentials: true

Python

gateway.py

from oculis import GatewayConfig, Cors

config = GatewayConfig(
    listen_address="0.0.0.0",
    listen_port=8080,
    metrics_port=9090,
    max_request_bytes=8 * 1024 * 1024,
    request_timeout_seconds=120,
    keepalive_seconds=75,
    cors=Cors(
        allowed_origins=["https://app.example.com"],
        allow_credentials=True,
    ),
)

keepalive_seconds must exceed the load balancer idle timeout

If the LB idle timeout is 60s and keepalive_seconds is 30s, the gateway closes connections the LB still considers live, and clients see intermittent resets during long streams. Set the gateway’s keepalive higher than the LB’s idle timeout, not lower.

Step 2 — Register upstreams

An upstream is a destination. Register every place a request could be sent, then let routes choose among them.

CLI / YAML

oculis-config.yaml

oculis:
  upstreams:
    # Self-hosted runtime
    - name: vllm-a
      provider: vllm
      endpoint: 'http://gpu-node-1:8000/v1'
      model: meta-llama/Llama-3.1-70B-Instruct
      weight: 1 # used by `weighted` strategy
      timeout_seconds: 30
      max_queue_depth: 50
      health_check:
        path: /health
        interval_seconds: 5
        timeout_seconds: 2
        healthy_threshold: 2
        unhealthy_threshold: 3

    - name: vllm-b
      provider: vllm
      endpoint: 'http://gpu-node-2:8000/v1'
      model: meta-llama/Llama-3.1-70B-Instruct
      timeout_seconds: 30
      max_queue_depth: 50

    # Commercial API
    - name: openai
      provider: openai
      model: gpt-4o-mini
      api_key: ${env:OPENAI_API_KEY}
      timeout_seconds: 60
      # Optional: override the base URL for Azure or a proxy
      # endpoint: "https://my-resource.openai.azure.com/openai/v1"

Python

gateway.py

from oculis import Upstream, HealthCheck

config.upstreams = [
    Upstream(
        name="vllm-a",
        provider="vllm",
        endpoint="http://gpu-node-1:8000/v1",
        model="meta-llama/Llama-3.1-70B-Instruct",
        weight=1,
        timeout_seconds=30,
        max_queue_depth=50,
        health_check=HealthCheck(
            path="/health",
            interval_seconds=5,
            timeout_seconds=2,
            healthy_threshold=2,
            unhealthy_threshold=3,
        ),
    ),
    Upstream(
        name="openai",
        provider="openai",
        model="gpt-4o-mini",
        api_key="${env:OPENAI_API_KEY}",
        timeout_seconds=60,
    ),
]

cURL

Register an upstream at runtime

curl -X PUT http://localhost:8080/admin/v1/upstreams/vllm-a \
  -H "Authorization: Bearer $OCULIS_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "provider": "vllm",
    "endpoint": "http://gpu-node-1:8000/v1",
    "model": "meta-llama/Llama-3.1-70B-Instruct",
    "timeout_seconds": 30,
    "max_queue_depth": 50
  }'

Supported providers

vllm · tgi · sglang · ollama · llamacpp · tensorrt · openai · azure_openai · anthropic · bedrock · vertex · mistral · together

Feature support varies between providers, so confirm behavior before putting two providers in one fallback chain.

Step 3 — Define routes

CLI / YAML

oculis-config.yaml

oculis:
  routes:
    # Even distribution across two identical GPU nodes.
    - name: default
      strategy: round_robin
      upstreams: [vllm-a, vllm-b]

    # Explicit alias so applications never name a model directly.
    - name: fast-summarize
      strategy: single
      primary: openai

    # Canary: 5% of traffic to a new node.
    - name: canary
      strategy: weighted
      upstreams:
        - name: vllm-a
          weight: 95
        - name: vllm-b
          weight: 5

Python

gateway.py

from oculis import Route, WeightedUpstream

config.routes = [
    Route(name="default", strategy="round_robin", upstreams=["vllm-a", "vllm-b"]),
    Route(name="fast-summarize", strategy="single", primary="openai"),
    Route(
        name="canary",
        strategy="weighted",
        upstreams=[
            WeightedUpstream(name="vllm-a", weight=95),
            WeightedUpstream(name="vllm-b", weight=5),
        ],
    ),
]

Choosing a strategy

Strategy	Distributes by	Choose it when
single	Nothing — one upstream	Development, or an explicit model alias.
round_robin	Even rotation over healthy upstreams	Nodes are identical and requests cost roughly the same.
least_queue	Current queue depth	Nodes differ in size, or request costs vary widely.
weighted	Fixed percentages	Canarying a node, model, or provider.
primary_with_fallback	Health and saturation	You own GPUs and want cloud as insurance.
semantic	Prompt classification	Cost reduction by sending easy prompts to a small model.

least_queue beats round_robin under mixed workloads

Round robin assumes requests are interchangeable. They are not: a 32K-token summarization occupies a node far longer than a 200-token classification. If your p99 is much worse than your p50, least_queue is usually the cheapest improvement available.

Step 4 — Apply

1. Validate.

   oculis config validate --file oculis-config.yaml --explain

2. Apply with a rolling restart.

   oculis apply --file oculis-config.yaml --strategy rolling --drain-timeout 60s

3. Watch until every replica reports the new generation.

   oculis status --watch

Verification

Confirms the listener is bound, every upstream is healthy, each route resolves, and load actually distributes across upstreams.

1. Every upstream is reachable and every route resolves.

oculis gateway test --all-routes

Expected output

[INFO]  Route "default"         strategy=round_robin   upstreams=2
[INFO]    vllm-a    http://gpu-node-1:8000/v1   OK  (latency 11ms, queue 0/50)
[INFO]    vllm-b    http://gpu-node-2:8000/v1   OK  (latency 13ms, queue 0/50)
[INFO]  Route "fast-summarize"  strategy=single        upstreams=1
[INFO]    openai    api.openai.com              OK  (latency 152ms)
[INFO]  Route "canary"          strategy=weighted      upstreams=2
[PASS]  3 routes resolved, 3 upstreams healthy, 0 unhealthy.

2. Traffic actually distributes. Send a burst and confirm the counters move on both upstreams:

for i in $(seq 1 10); do
  curl -s -o /dev/null http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"default","messages":[{"role":"user","content":"hi"}],"max_tokens":4}'
done

curl -s http://localhost:9090/metrics | grep 'oculis_requests_total{route="default"'

Expected output

oculis_requests_total{route="default",upstream="vllm-a",status="200"} 5
oculis_requests_total{route="default",upstream="vllm-b",status="200"} 5

If all ten landed on one upstream, the other is failing its health check — it will be reported as unhealthy in step 1 rather than silently skipped.

Model aliasing

Applications send a route name in the model field. This is what lets you change models without shipping application code:

# The application asks for a route...
curl http://localhost:8080/v1/chat/completions \
  -d '{"model": "fast-summarize", ...}'

# ...and you decide what that means.
routes:
  - name: fast-summarize
    strategy: single
    primary: openai # change to vllm-a; no application change

To let clients continue sending real model names during a migration, add explicit aliases:

oculis:
  aliases:
    'gpt-4o-mini': fast-summarize
    'gpt-4o': high-quality

Health checks

Field	Default	Guidance
path	/health	Must return 200 without authentication.
interval_seconds	10	Lower means faster detection and more probe load.
timeout_seconds	2	Should be well under interval_seconds.
healthy_threshold	2	Consecutive successes before returning to rotation.
unhealthy_threshold	3	Below 3, a single GC pause evicts a healthy node.

An authenticated health path takes your whole tier down

If /health sits behind auth, every probe returns 401, every node is marked unhealthy, and the router reports ERR_OCULIS_NO_HEALTHY_UPSTREAM while the nodes are perfectly fine. Verify the path is anonymous: curl -i http://gpu-node-1:8000/health.

Edge cases and known limitations

• max_queue_depth is per replica. Three replicas at depth 50 tolerate 150 queued requests.
• Weighted routing is probabilistic, not exact. Over 100 requests a 95/5 split may land 93/7. Judge a canary over thousands of requests, not dozens.
• Changing a route’s strategy does not drain in-flight requests. They complete against the upstream they were already assigned.
• request_timeout_seconds is a hard ceiling that overrides any longer per-upstream timeout. A 60s upstream timeout under a 30s gateway timeout is effectively 30s.
• Health checks do not validate the model. A node serving the wrong model passes /health happily. Use oculis gateway test to check the model identity.

Next steps

Fallback & failoverCircuit breakers, overflow routing, and provider failover.

Authentication & RBACStop running an unauthenticated listener.

Rate limits & token budgetsProtect the tier you just built.