Networking & Auth

Applies tov1.4.0DifficultyIntermediate

These failures happen before any inference work begins, which is good news: they are cheap to reproduce and the error messages are usually precise.

Error code	Symptom & log signature	Root cause & first action
ErrorERR_OCULIS_UPSTREAM_TLS_VERIFYHTTP 502	Only self-hosted or on-prem upstreams fail; public providers work. x509: certificate signed by unknown authority	A private or intercepting CA is not present in the agent trust store. FixMount the CA bundle and set `tls.ca_bundle_path`. Never disable verification in production. Full procedure
ErrorERR_OCULIS_AUTH_INVALID_KEYHTTP 401	Every request from one client fails while others succeed. auth: key id=oc_live_3f2a… not found or revoked	Key revoked, rotated, or issued in a different environment than the one called. FixReissue the key and confirm the client targets the matching environment. Full procedure
ErrorERR_OCULIS_AUTH_SCOPE_DENIEDHTTP 403	Authentication succeeds but specific models or routes are refused. rbac: key lacks scope models:invoke for target gpt-4o	The API key’s scope set excludes the requested model or operation. FixGrant the scope on the key, or route the client to a permitted model alias. Full procedure
WarningERR_OCULIS_BUDGET_EXCEEDEDHTTP 402	A tenant is cut off mid-month while other tenants are unaffected. budget: tenant acme consumed 100.4% of monthly cap ($2,010/$2,000)	The tenant reached its configured spend ceiling. Working as designed. FixRaise the cap, or configure a downgrade route to a cheaper model at the ceiling. Full procedure
WarningERR_OCULIS_RATE_LIMIT_LOCALHTTP 429	Clients are throttled by Oculis itself rather than by the upstream provider. ratelimit: tenant acme exceeded 600 rpm (bucket=tenant:acme)	The tenant exceeded its configured request or token rate. FixRaise the limit, widen the burst allowance, or move the client to its own bucket. Full procedure

Isolating the failing hop

There are four hops, and each has a different owner. Test them in order.

  client ──1──► load balancer ──2──► Oculis ──3──► upstream provider
                                        │
                                        └──4──► vector store / guardrail sidecar

1. Client to load balancer.

   curl -sv https://gateway.example.com/healthz 2>&1 | grep -E '^[*<>]' | head -20

2. Load balancer to Oculis. From inside the network:

   curl -s http://oculis.internal:8080/healthz

3. Oculis to upstream. Use the agent’s own view, which uses the same trust store and proxy settings as real traffic:

   oculis upstream test --name openai --verbose

4. Oculis to auxiliary services.

   oculis vector test --store docs
   oculis guardrail test --policy pii-strict --input "ping"

Test from inside the pod, not from your laptop

curl from a workstation traverses a different network path, different DNS, and a different trust store. A test that passes from your laptop and fails from the pod has told you exactly where the problem is — use kubectl exec or docker exec.

TLS verification failures

Signature: ERR_OCULIS_UPSTREAM_TLS_VERIFY

x509: certificate signed by unknown authority

Public providers work; private or on-prem upstreams fail. A private CA — or a corporate TLS inspection proxy — is not in the agent’s trust store.

Diagnose

openssl s_client -connect vllm.internal:8443 -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer -subject -dates

If the issuer is your internal CA (or something named like a security appliance), that is the answer.

Resolve

CLI / YAML

oculis-config.yaml

oculis:
  tls:
    ca_bundle_path: /etc/ssl/certs/internal-ca.pem
    min_version: '1.2'
    # For mutual TLS to an upstream:
    client_cert_path: /etc/oculis/tls/client.crt
    client_key_path: /etc/oculis/tls/client.key

Helm

values.yaml

tls:
  caBundle:
    configMapRef:
      name: internal-ca
      key: ca.pem
  minVersion: '1.2'

extraVolumeMounts:
  - name: internal-ca
    mountPath: /etc/ssl/certs/internal-ca.pem
    subPath: ca.pem
    readOnly: true

Docker

docker run -d --name oculis \
  -v /etc/ssl/certs/internal-ca.pem:/etc/ssl/certs/internal-ca.pem:ro \
  -e OCULIS_TLS_CA_BUNDLE=/etc/ssl/certs/internal-ca.pem \
  ghcr.io/selaware/oculis-agent:1.4.0

Never disable verification in production

insecure_skip_verify: true exists for local development and makes every upstream connection trivially interceptable. If you are tempted to set it in production, mount the CA bundle instead — it takes about the same amount of time.

Expired certificates

x509: certificate has expired or is not yet valid: current time ... is after ...

Also check clock skew. A node whose clock is days ahead will reject valid certificates:

timedatectl status | grep -E 'synchronized|Time zone'

DNS and connectivity

dial tcp: lookup vllm.inference.svc.cluster.local: no such host

# Resolution
getent hosts vllm.inference.svc.cluster.local

# Reachability, without TLS in the way
nc -zv vllm.inference.svc.cluster.local 8000

# Kubernetes: does the Service have endpoints?
kubectl get endpoints -n inference vllm

An empty ENDPOINTS column means no pods match the Service selector — the Service exists, but points at nothing. That surfaces as a DNS or connection failure rather than as an obvious misconfiguration.

Egress restrictions

In a locked-down cluster, egress to cloud providers is frequently the missing piece:

networkpolicy.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: oculis-egress
  namespace: oculis
spec:
  podSelector:
    matchLabels: { app.kubernetes.io/name: oculis-agent }
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: inference }
      ports:
        - { protocol: TCP, port: 8000 }
    - ports: # DNS
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
    - ports: # HTTPS to cloud providers
        - { protocol: TCP, port: 443 }

If you use an HTTP proxy for egress, tell the agent explicitly:

oculis:
  http_proxy: 'http://proxy.internal:3128'
  no_proxy: '.svc.cluster.local,10.0.0.0/8,localhost'

Authentication failures

401 — identity

Signature: ERR_OCULIS_AUTH_INVALID_KEY

auth: key id=oc_live_3f2a… not found or revoked

The key was not recognized. Check, in order:

oculis keys inspect --prefix oc_live_3f2a

Example output

  id         key_01HQ8Z3K4M5N6P7Q8R9S
  tenant     acme
  status     REVOKED            ← here it is
  revoked    2026-07-14T09:12:00Z  reason="rotated"
  scopes     models:invoke, models:list

Common causes:

• Revoked or rotated, and the client still holds the old value.
• Expired. Expiry returns 401, not 403.
• Wrong environment. A staging key against production is simply unknown.
• Header mangling. A proxy stripping or rewriting Authorization.

# Confirm the header survives the full path:
curl -s https://gateway.example.com/v1/debug/echo-headers \
  -H "Authorization: Bearer $OCULIS_KEY" | grep -i authorization

403 — permissions

Signature: ERR_OCULIS_AUTH_SCOPE_DENIED

rbac: key lacks scope models:invoke for target gpt-4o

The key is valid but not permitted. This is a different problem from 401 and needs a different fix:

oculis keys test --id key_01HQ8Z3K4M5N6P7Q8R9S --scope models:invoke --route high-quality

Grant the scope or the route.

Revocation is not instant across replicas

Each replica caches key lookups for auth.cache_seconds (default 30). After revoking a key, requests may succeed on some replicas for up to that long. For immediate revocation, use a shared key store with cache_seconds: 0.

402 — budget

A ERR_OCULIS_BUDGET_EXCEEDED is not an auth failure, even though it looks like one. The key is valid and permitted; the tenant is out of budget. See token and spend budgets.

Verification

Confirms every hop resolves, TLS validates against the configured trust store, and auth returns the right status for valid, invalid, and under-scoped keys.

1. All upstream connections are healthy from inside the pod.

oculis upstream test --all --verbose

Expected output

  vllm-primary   http://vllm.inference.svc:8000/v1
    dns          OK   10.4.2.11
    tcp          OK   3ms
    tls          --   plaintext (in-cluster)
    http         OK   200 (11ms)
  openai         https://api.openai.com/v1
    dns          OK   104.18.7.192
    tcp          OK   14ms
    tls          OK   verified by DigiCert Global Root G2 (expires 2031-11-09)
    http         OK   200 (152ms)
[PASS]  2/2 upstreams reachable.

2. Auth returns the correct status for each case.

oculis auth test --key "$OCULIS_KEY" --expect 200 && \
oculis auth test --key "oc_live_invalid" --expect 401 && \
oculis auth test --key "$OCULIS_KEY" --route high-quality --expect 403

Expected output

[PASS]  valid key            -> 200
[PASS]  unknown key          -> 401 ERR_OCULIS_AUTH_INVALID_KEY
[PASS]  key without route    -> 403 ERR_OCULIS_AUTH_SCOPE_DENIED

All three must pass. A configuration that returns 200 for the third case is not enforcing route restrictions, regardless of what the config file says.

Next steps

Authentication & RBACKeys, scopes, roles, and OIDC configuration.

Inference & streamingIf connectivity is fine but responses misbehave.

Collecting diagnosticsBundle connectivity and TLS state for escalation.