Benchmarking Methodology

Applies tov1.4.0DifficultyIntermediateImpactNo restart

Read this before the other optimization pages. Every tuning change trades something, and without a baseline you cannot tell an improvement from a redistribution.

Prerequisites

• Oculis Collector v1.4.0 or later
• Telemetry configured — the benchmark reads the same metrics
• A load generation host that is not the GPU node, with headroom to saturate the target
• A prompt corpus representative of production traffic (see below)
• An environment you can saturate. Do not benchmark against live production traffic.

Verify all of the aboveoculis status --check-deps

Four rules

1. Benchmark with your traffic, not with a synthetic profile. Prompt length distribution determines prefill cost, and output length determines decode cost. A benchmark using 128-token prompts tells you nothing about a RAG workload with 4,000-token prompts.

2. Measure at fixed concurrency, not fixed rate — or both, deliberately. Fixed-rate tests degenerate once you exceed capacity: the queue grows without bound and every number after that describes queueing, not the system.

3. Report percentiles, never means. Mean latency hides the tail users actually complain about. Report p50, p95, p99.

4. Change one thing at a time. Enabling prefix caching and FP8 KV cache together and observing an improvement tells you nothing about which one did it, or whether one hurt.

Warm up before you measure

The first requests after a restart pay CUDA graph capture, cache population, and JIT compilation. Discard at least 60 seconds of warm-up, or your “before” is artificially bad and every change looks like a win.

Build a representative corpus

Sample from real traffic rather than inventing prompts:

oculis bench corpus export \
  --route default \
  --since 7d \
  --sample 2000 \
  --stratify-by prompt_tokens \
  --redact \
  --output corpus.jsonl

--stratify-by prompt_tokens preserves the length distribution, which is what makes the benchmark predictive. --redact strips PII so the corpus can live in a repository.

Check that it looks like production:

oculis bench corpus stats --file corpus.jsonl

Example output

  samples                2,000
  prompt tokens    p50 1,284   p95 6,912   p99 14,208   max 31,884
  output tokens    p50   204   p95   892   p99  1,904   max  4,096
  system prefix    shared by 94.2% of samples (1,182 tokens)
  streaming              88.1% of samples

That system prefix shared by 94.2% line is worth noticing — it predicts a large win from prefix caching before you enable anything.

Establish a baseline

1. Pin the environment. Record the agent version, engine config, model revision, driver, and GPU. A baseline without provenance is not comparable to anything later.

   oculis bench env capture --output baseline-env.json

2. Warm up, then measure.

   oculis bench run \
     --route default \
     --corpus corpus.jsonl \
     --concurrency 32 \
     --warmup 60s \
     --duration 300s \
     --output baseline.json

3. Run it three times and confirm the runs agree within a few percent. A benchmark that is not reproducible cannot detect a 10% improvement.

   oculis bench run --corpus corpus.jsonl --repeat 3 --output baseline.json

   Example output

     run    ttft_p95   tpot_p50   throughput   errors
       1     2,841ms       24ms    412 tok/s        0
       2     2,798ms       24ms    418 tok/s        0
       3     2,864ms       25ms    409 tok/s        0
     ---
     cv        1.2%        2.4%         1.1%
   [PASS] Coefficient of variation below 5% — baseline is reproducible.

4. Commit the baseline to your repository next to the config it describes.

Find the saturation point

A single concurrency number tells you one point on a curve. The shape is what you need:

oculis bench sweep \
  --route default \
  --corpus corpus.jsonl \
  --concurrency 1,2,4,8,16,32,64,128 \
  --duration 120s \
  --output sweep.json

Example output

  conc    ttft_p95    tpot_p50   throughput   queue_p95   preempt
     1       182ms        18ms     48 tok/s           0         0
     2       194ms        18ms     94 tok/s           0         0
     4       221ms        19ms    186 tok/s           0         0
     8       288ms        20ms    342 tok/s           1         0
    16       502ms        22ms    398 tok/s           4         0
    32     2,841ms        24ms    412 tok/s          19         0   ← knee
    64     8,912ms        41ms    408 tok/s          58     2,104
   128    24,102ms        88ms    331 tok/s         119    18,882

[INFO]  Knee at concurrency 16–32. Throughput plateaus at ~412 tok/s.
[WARN]  Above 64, preemption begins and throughput regresses.

Read three things from this:

• The knee (~16–32) is your real operating capacity. Size max_queue_depth and rate limits around it, not around the maximum number the hardware will accept.
• The plateau (~412 tok/s) is the ceiling. More concurrency does not buy throughput.
• The regression above 64 is preemption. Past that point you are actively losing work — see KV cache exhaustion.

Compare a change

Change exactly one thing, then:

oculis bench run \
  --route default \
  --corpus corpus.jsonl \
  --concurrency 32 \
  --warmup 60s \
  --duration 300s \
  --compare baseline.json \
  --output prefix-cache-enabled.json

Expected output

                        baseline      current     delta    significant
  ttft p50                842ms        291ms    -65.4%            yes
  ttft p95              2,841ms        712ms    -74.9%            yes
  tpot p50                 24ms         23ms     -4.2%             no
  throughput            412 tok/s    689 tok/s   +67.2%           yes
  errors                      0            0         —              —

[PASS]  1 change under test: engine.enable_prefix_caching=true

The significant column compares the delta against the run-to-run variance you established earlier. A 4% TPOT change against 2.4% variance is noise, and labeling it as such stops teams from attributing improvements to the wrong setting.

Metrics worth reporting

Metric	Definition	Why it matters
TTFT p50 / p95	Time to first token	Perceived responsiveness
TPOT p50	Time per output token after the first	Perceived generation speed
Throughput	Output tokens per second, all requests	Capacity and cost per token
Requests/sec	Completed requests per second	Capacity in application terms
Queue depth p95	Requests waiting	Saturation headroom
Preemptions	Sequences evicted and recomputed	Over-capacity signal, before errors appear
Error rate	Non-2xx responses	The number that invalidates every other one
Cost per 1k req	Estimated spend	The reason for most tuning

An improvement with a raised error rate is not an improvement

Shedding load makes every latency percentile look better. Always report error rate alongside latency, and treat a change that improves p95 while raising errors as a regression.

Continuous regression testing

Run the benchmark in CI against a fixed corpus so performance regressions are caught in review rather than in production:

.github/workflows/perf.yml

name: Performance regression

on:
  pull_request:
    paths:
      - 'config/**'
      - 'charts/**'

jobs:
  bench:
    runs-on: [self-hosted, gpu]
    steps:
      - uses: actions/checkout@v5

      - name: Run benchmark
        run: |
          oculis bench run \
            --corpus tests/corpus.jsonl \
            --concurrency 32 \
            --warmup 60s \
            --duration 180s \
            --compare tests/baseline.json \
            --fail-on-regression ttft_p95:10%,throughput:-10%,errors:0 \
            --output result.json

      - name: Upload result
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: bench-result
          path: result.json

--fail-on-regression fails the job when p95 TTFT worsens by more than 10%, throughput drops more than 10%, or any error appears.

Verification

Confirms the benchmark itself is trustworthy before you draw conclusions from it: reproducible across runs, and not bottlenecked on the load generator.

1. Reproducibility.

oculis bench run --corpus corpus.jsonl --concurrency 32 --repeat 3 --report-variance

Expected output

  metric        mean      stdev       cv
  ttft_p95   2,834ms       33ms     1.2%
  tpot_p50       24ms      0.6ms     2.4%
  throughput  413 tok/s   4.5 tok/s  1.1%
[PASS]  All coefficients of variation below 5%.

A CV above ~5% means something else is moving — noisy neighbors, thermal throttling, or a load generator that cannot keep up. Fix that before benchmarking anything.

2. The load generator is not the bottleneck.

oculis bench doctor --concurrency 32

Expected output

  generator cpu       38% (headroom OK)
  generator network   112 Mbps of 10 Gbps (headroom OK)
  open file limit     1,048,576 (OK)
  clock sync          offset 0.4ms (OK)
  target reachable    3ms RTT
[PASS]  Load generator is not the constraint.

If generator CPU is above ~80%, your benchmark is measuring the client.

Edge cases and known limitations

• Cloud provider benchmarks are not reproducible. Shared multi-tenant capacity means results vary by time of day. Run longer, repeat more, and treat absolute numbers as indicative.
• Prefix caching inflates repeated-corpus benchmarks. Running the same 2,000 prompts repeatedly produces a cache hit ratio no production workload will see. Use --no-repeat-corpus for a conservative number.
• Thermal throttling appears in long runs. A 10-minute benchmark can end slower than it began on air-cooled hardware. Check nvidia-smi --query-gpu=clocks_throttle_reasons.active.
• Benchmarks do not measure quality. Pair every performance run with oculis eval run — a faster configuration that answers worse is not a better configuration.

Next steps

Latency tuningNow that you have a baseline, reduce TTFT and TPOT.

Throughput & costBatching, speculative decoding, and semantic routing.

Capacity planningTurn the saturation point into a provisioning decision.