Collecting Diagnostics

Applies tov1.4.0DifficultyBeginnerImpactNo restart

When the diagnostic matrix does not resolve it, collect a bundle before you escalate. A bundle turns a multi-day back-and-forth into a single exchange.

Capture state before you restart

A restart clears queue depth, circuit-breaker state, KV cache utilization, and in-memory counters — exactly the evidence that explains what happened. Run the bundle command first, then restart.

Generate a bundle

oculis support bundle --output oculis-bundle-$(date +%Y%m%d-%H%M).tar.gz

Expected output

Collecting diagnostics…
  ✓ agent version and build metadata
  ✓ resolved configuration (secrets redacted)
  ✓ upstream health and circuit breaker state
  ✓ queue depth and KV cache utilization
  ✓ last 10,000 log lines (redacted)
  ✓ metrics snapshot
  ✓ GPU inventory and Xid history
  ✓ 20 slowest and 20 most recent failed traces
  ✗ vector store stats (skipped: --include-vector not set)

Wrote oculis-bundle-20260807-0921.tar.gz (4.2 MB)
Redacted 14 secrets, 3 API keys, 1,204 prompt bodies.

Scope it to a specific incident when you can:

oculis support bundle \
  --since 2h \
  --request-id req_01HQ8Z3K4M5N6P7Q8R9S \
  --include-vector \
  --include-gpu-dump \
  --output incident-4471.tar.gz

What is in it

• Directoryoculis-bundle-20260807-0921/

  • manifest.json bundle version, redaction summary, collection time
  • Directoryagent/

    • version.txt
    • config-resolved.yaml secrets redacted
    • config-generation-history.json
  • Directoryruntime/

    • upstream-health.json
    • breaker-state.json
    • queue-depth.json
    • kv-cache.json
  • Directorygpu/

    • nvidia-smi.txt
    • xid-history.txt
    • topology.txt
  • Directorylogs/

    • agent.log last 10,000 lines, redacted
  • Directorymetrics/

    • snapshot.prom
  • Directorytraces/

    • slowest-20.json
    • failed-20.json

What is redacted

Redaction is on by default and cannot be silently disabled.

Redacted	Retained
API keys, tokens, passwords	Key IDs and tenant names
Authorization headers	Request IDs and trace IDs
Prompt and completion bodies	Token counts, model names, timings
Retrieved document content	Chunk IDs, scores, collection names
Connection strings and DSNs	Hostnames and ports
Detected PII matches	Detector name and offset

Verify before sending:

oculis support verify --file oculis-bundle-20260807-0921.tar.gz

Expected output

  bundle version    1
  redaction         ENABLED
  secrets found     0
  prompt bodies     0
  high-entropy strings 0
[PASS]  Bundle is safe to share.

Including prompt bodies is a deliberate act

--include-prompts exists for cases where the content is the bug — a tokenization issue, a guardrail false positive. It disables body redaction. Only use it with data you are permitted to share, and say so explicitly when you send the bundle.

Gather the context the bundle cannot

The bundle captures machine state. These four questions capture what a human knows, and they are what actually speeds up diagnosis:

1. What changed? A deploy, a config change, a traffic shift, a provider incident, a driver update. “Nothing changed” is almost never true — check the config generation history in the bundle.

   oculis config history --limit 10

2. When did it start, precisely? A timestamp lets everyone align logs, metrics, and traces. “Since yesterday” does not.

3. What is the blast radius? One tenant, one route, one replica, one GPU node — or everything? Narrow scope points directly at the cause.

   oculis status --by-replica

4. One failing request ID. A single concrete example beats any aggregate description.

   oculis trace get req_01HQ8Z3K4M5N6P7Q8R9S --format tree

Live diagnostics

For a problem happening right now, these are safe to run against production:

# Watch queue depth and breaker state, refreshed every second
oculis status --watch --interval 1s

# Tail logs for one route only, without raising the global log level
oculis logs tail --route default --level debug

# Sample in-flight requests
oculis requests list --state in_flight --limit 20

Scope a debug log level to one route rather than globally, and remove it afterwards:

oculis config set observability.log_overrides.route:default=debug --reload
# …reproduce…
oculis config unset observability.log_overrides.route:default --reload

Debug logging on a busy gateway buries the signal

At production volume, global debug logging can produce gigabytes per minute and slow the agent enough to change the behavior you are investigating. Always scope it.

Verification

Confirms the bundle was produced, contains the expected sections, and passes redaction verification before it leaves your environment.

oculis support verify --file oculis-bundle-20260807-0921.tar.gz --list-sections

Expected output

  agent/          4 files
  runtime/        4 files
  gpu/            3 files
  logs/           1 file  (10,000 lines)
  metrics/        1 file
  traces/         2 files (40 traces)

  redaction       ENABLED
  secrets found   0
[PASS]  Bundle complete and safe to share.

If secrets found is non-zero, do not send the bundle. Re-run generation and open an issue — that is a redaction bug and we want to know about it.

Where to send it

Send the bundle to your SelaWare support contact, along with the four answers above and one failing request ID. That is enough to start.

Next steps

Diagnostic matrixSearch by symptom, log line, or error code before escalating.

Telemetry & observabilityInstrument now so the next incident already has evidence.