Telemetry & Observability

Applies tov1.4.0DifficultyIntermediateImpactNo restart

Oculis emits OpenTelemetry traces and Prometheus metrics. It does not store them — point it at the backend you already run.

The value is in what it measures: per-hop timing inside a request (guardrail, retrieval, queue, prefill, decode), and per-request cost attributed to a tenant.

Prerequisites

• Oculis Collector v1.4.0 or later
• An OTLP endpoint — the OpenTelemetry Collector, or a vendor’s OTLP intake
• A Prometheus-compatible scraper with network access to :9090
• Network: egress to the OTLP endpoint (4317 gRPC or 4318 HTTP)
• Permissions: reload access is enough — telemetry changes do not require a restart

Verify all of the aboveoculis status --check-deps

Step 1 — Export traces

CLI / YAML

oculis-config.yaml

oculis:
  observability:
    log_level: info
    log_format: json # use `text` only for local development

    tracing:
      enabled: true
      exporter: otlp
      endpoint: 'http://otel-collector.observability.svc:4317'
      protocol: grpc # grpc | http/protobuf
      sample_rate: 0.1 # 10% of requests
      # Always trace failures and slow requests regardless of sample rate.
      always_sample:
        - errors
        - slower_than_ms: 5000
      resource_attributes:
        service.name: oculis-gateway
        deployment.environment: production

Helm

values.yaml

observability:
  logLevel: info
  logFormat: json
  tracing:
    enabled: true
    exporter: otlp
    endpoint: http://otel-collector.observability.svc:4317
    protocol: grpc
    sampleRate: 0.1
    alwaysSample:
      - errors
      - slowerThanMs: 5000
    resourceAttributes:
      service.name: oculis-gateway
      deployment.environment: production

Python

telemetry.py

from oculis import Observability, Tracing

config.observability = Observability(
    log_level="info",
    log_format="json",
    tracing=Tracing(
        enabled=True,
        exporter="otlp",
        endpoint="http://otel-collector.observability.svc:4317",
        protocol="grpc",
        sample_rate=0.1,
        always_sample=["errors", {"slower_than_ms": 5000}],
        resource_attributes={
            "service.name": "oculis-gateway",
            "deployment.environment": "production",
        },
    ),
)

Tail-based sampling on errors is the setting that pays for itself

At sample_rate: 0.1 you keep 10% of traces — and statistically miss 90% of the rare failures you actually need. always_sample: [errors, slower_than_ms] keeps every trace worth having while still discarding the bulk of healthy traffic.

Span structure

A single request produces one trace with a predictable shape, which is what makes oculis trace get useful during an incident:

oculis.request                                        1,847ms
├── oculis.auth                                           2ms
├── oculis.ratelimit                                      1ms
├── oculis.guardrail.request         policy=pii-strict   38ms
├── oculis.retrieval                 store=docs          94ms
│   ├── oculis.embed                 model=text-emb-3    61ms
│   └── oculis.vector.search         k=8                 11ms
├── oculis.context.assemble          tokens=3891          3ms
├── oculis.route                     strategy=p_w_f       1ms
├── oculis.queue.wait                depth=12           412ms   ← the real problem
└── oculis.upstream                  name=vllm-primary 1,290ms
    ├── oculis.prefill               tokens=3891        380ms
    └── oculis.decode                tokens=512         910ms

When someone reports “the model is slow”, this is how you show them 412 ms of it was queueing and not the model at all.

Step 2 — Export metrics

oculis-config.yaml

oculis:
  observability:
    metrics:
      enabled: true
      exporter: prometheus
      port: 9090
      path: /metrics
      # High-cardinality labels are opt-in; `tenant` is usually worth it.
      labels:
        include: [route, upstream, tenant, status]
        exclude: [request_id, api_key_id] # never label by these

Never label metrics by request ID or API key

Each distinct label value creates a new time series. Labeling by request_id produces one series per request and will take down your Prometheus, not just make it slow. The exclude list above is a guardrail against that, and it is deliberate.

Metrics worth alerting on

Metric	Type	Why it matters
oculis_requests_total	counter	Baseline volume, by route, upstream, and status.
oculis_request_duration_seconds	histogram	End-to-end latency. Alert on p95, not mean.
oculis_ttft_seconds	histogram	Perceived responsiveness. The leading indicator for users.
oculis_queue_depth	gauge	Saturation. Rising queue depth precedes every latency incident.
oculis_breaker_state	gauge	1 when open. Any open breaker deserves attention.
oculis_overflow_total	counter	Requests spilled to fallback. Rising means the primary is sick.
oculis_upstream_errors_total	counter	By upstream and error code.
oculis_tokens_total	counter	Input and output tokens, by tenant and model.
oculis_cost_usd_total	counter	Estimated spend, by tenant.
oculis_guardrail_actions_total	counter	Blocks and redactions, by policy and detector.
oculis_retrieval_fail_open_total	counter	Ungrounded answers served during a store outage.
oculis_kv_cache_utilization	gauge	Preemption risk. Sustained >0.9 precedes throughput collapse.

Starter alert rules

alerts.yaml

groups:
  - name: oculis
    rules:
      - alert: OculisTTFTDegraded
        expr: histogram_quantile(0.95, sum(rate(oculis_ttft_seconds_bucket[5m])) by (le, route)) > 1.5
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: 'p95 TTFT above 1.5s on route {{ $labels.route }}'
          runbook: 'https://docs.selaware.ai/ai/performance-tuning/#reducing-time-to-first-token'

      - alert: OculisBreakerOpen
        expr: max(oculis_breaker_state{state="open"}) by (upstream) == 1
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: 'Circuit breaker open for {{ $labels.upstream }}'
          runbook: 'https://docs.selaware.ai/ai/gateway-routing/fallback-failover/#circuit-breaker-behavior'

      - alert: OculisNoHealthyUpstream
        expr: sum(oculis_upstreams_healthy) by (route) == 0
        for: 1m
        labels: { severity: critical }
        annotations:
          summary: 'No healthy upstream for route {{ $labels.route }}'
          runbook: 'https://docs.selaware.ai/oculis/reference/error-index/#err-oculis-no-healthy-upstream'

      - alert: OculisKVCachePressure
        expr: avg(oculis_kv_cache_utilization) by (upstream) > 0.9
        for: 15m
        labels: { severity: warning }
        annotations:
          summary: 'KV cache above 90% on {{ $labels.upstream }} — preemption likely'
          runbook: 'https://docs.selaware.ai/ai/capacity-planning/'

      - alert: OculisRetrievalSilentlyDegraded
        expr: rate(oculis_retrieval_fail_open_total[10m]) > 0
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: 'Serving ungrounded answers — vector store unreachable'
          runbook: 'https://docs.selaware.ai/ai/rag-vector-databases/#connection-health-and-failure-modes'

That last rule catches a failure that is otherwise invisible: retrieval failing open produces fluent, confident, ungrounded answers rather than errors.

Step 3 — Cost attribution

oculis-config.yaml

oculis:
  observability:
    cost_tracking:
      enabled: true
      attribute_by: [tenant, route, model]
      emit_per_request: true # adds cost as a span attribute

oculis cost report --since 30d --group-by tenant --format table

Example output

tenant            requests    input_tokens   output_tokens   est_cost_usd
acme               418,204     512,884,102      41,002,884       1,568.82
globex             102,881     118,204,881       9,884,201         392.14
internal-batch   1,204,882   2,884,102,884     102,884,102          0.00*
                 ---------   -------------   -------------   ------------
total            1,725,967   3,515,191,867     153,771,187       1,960.96

* self-hosted; see pricing.infra_cost_per_hour_usd for amortized attribution

Cost figures are estimates

Costs are derived from token counts and your configured price table. Provider discounts, prompt caching credits, and batch pricing are not reflected. If your counts drift from provider billing, check ERR_OCULIS_TOKENIZER_MISMATCH.

Verification

Confirms metrics are scrapeable, traces reach the collector with the expected span structure, and cost attribution is recording. Check the trace end to end — an exporter that silently drops spans looks identical to one that works.

1. Metrics are exposed and populated.

curl -s http://localhost:9090/metrics | grep -c '^oculis_'

Expected output

47

A 0 here means the exporter is disabled or bound to a different interface.

2. A trace actually reaches the collector.

oculis trace test --emit-sample

Expected output

[INFO]  Emitting synthetic trace to http://otel-collector.observability.svc:4317
[INFO]  trace_id=4bf92f3577b34da6a3ce929d0e0e4736
[INFO]  Exporter acknowledged 11 spans in 34ms
[PASS]  OTLP export verified end to end.

3. The per-hop breakdown resolves for a real request.

oculis trace get req_01HQ8Z3K4M5N6P7Q8R9S --format tree

The output should match the span structure shown above. If oculis.queue.wait is missing, tracing is enabled but the router is not instrumented — check that sample_rate did not exclude this request.

4. Cost tracking is recording.

curl -s http://localhost:9090/metrics | grep oculis_cost_usd_total

Expected output

oculis_cost_usd_total{tenant="acme",route="default",model="gpt-4o-mini"} 12.4081

Edge cases and known limitations

• Telemetry export never blocks a response. Spans are buffered and dropped if the exporter falls behind. Watch oculis_telemetry_dropped_total — silent drops make traces misleading.
• Sampling is per trace, not per span. A sampled request keeps all its spans; an unsampled one keeps none. There are no partial traces.
• log_format: text is not machine-parseable. Use json anywhere logs are shipped.
• Streaming requests report duration at stream close, so a long stream appears as one long span rather than incremental progress. TTFT is recorded separately for exactly this reason.
• Cost is computed at response time using the price table loaded then. Changing prices does not retroactively restate history.

Next steps

Benchmarking methodologyNow that you can measure, establish a baseline before tuning.

Latency tuningAct on what the per-hop breakdown shows you.

Collecting diagnosticsBundle traces, config, and metrics for escalation.