Throughput & Cost

Applies tov1.4.0HardwareNVIDIA A100 / H100DifficultyAdvancedImpactRolling restart

Throughput and cost are the same problem viewed from two sides. More tokens per GPU-second means fewer GPUs, and fewer requests sent to an expensive model means a smaller invoice.

Ranked by return on effort, for most teams:

1. Semantic routing — send easy requests to a small model. Often 40–70% cost reduction.
2. Prompt caching — stop paying for the same prefix repeatedly.
3. Batching tuned to your traffic — the default is rarely right for your shape.
4. Speculative decoding — real gains, real complexity.

Prerequisites

• Oculis Collector v1.4.0 or later
• A benchmark baseline — mandatory here; every change on this page trades something
• Cost tracking enabled to measure spend
• For speculative decoding: a compatible draft model sharing the target’s tokenizer, plus VRAM for both
• Permissions: rolling restart of the inference tier

Verify all of the aboveoculis status --check-deps

Semantic routing

Most production traffic is not uniformly hard. Classification, extraction, and short-form summarization do not need your largest model — but they get it, because routing is static.

CLI / YAML

oculis-config.yaml

oculis:
  routes:
    - name: adaptive
      strategy: semantic
      classifier:
        model: oculis/router-small-v2
        cache_ttl_seconds: 3600 # identical prompts classify once
      tiers:
        - name: simple
          match:
            complexity_below: 0.35
            max_prompt_tokens: 2000
          upstream: llama-8b # cheap and fast

        - name: standard
          match:
            complexity_below: 0.75
          upstream: llama-70b

        - name: complex
          match: {} # everything else
          upstream: openai-gpt4o

      # Escalate rather than answer badly.
      escalation:
        enabled: true
        on_low_confidence: true
        confidence_threshold: 0.6

Python

routing.py

from oculis import Route, SemanticTier, Classifier, Escalation

route = Route(
    name="adaptive",
    strategy="semantic",
    classifier=Classifier(model="oculis/router-small-v2", cache_ttl_seconds=3600),
    tiers=[
        SemanticTier(name="simple", complexity_below=0.35,
                     max_prompt_tokens=2000, upstream="llama-8b"),
        SemanticTier(name="standard", complexity_below=0.75, upstream="llama-70b"),
        SemanticTier(name="complex", upstream="openai-gpt4o"),
    ],
    escalation=Escalation(enabled=True, on_low_confidence=True, confidence_threshold=0.6),
)

Measure the distribution before you trust it:

oculis route analyze --route adaptive --since 7d

Example output

  tier        requests    share    avg_cost    est_monthly
  simple       412,884    58.2%     $0.0002       $  247.73
  standard     241,002    34.0%     $0.0011       $  795.31
  complex       55,204     7.8%     $0.0140       $2,318.57
                                                  ----------
  total                                           $3,361.61
  all-complex baseline                            $9,940.28
  saving                                          $6,578.67 (66.2%)

Validate quality per tier, not in aggregate

An aggregate quality score hides the failure mode that matters: requests misrouted to the small tier. Evaluate each tier against a held-out set and watch the escalation rate. A rising escalation rate means the classifier is drifting relative to your traffic.

Prompt caching

For cloud providers that support it, this is the cheapest available saving — you are billed less for cached prefix tokens.

oculis-config.yaml

oculis:
  upstreams:
    - name: openai-gpt4o
      provider: openai
      prompt_cache:
        enabled: true
        min_prefix_tokens: 1024 # provider minimum
    - name: anthropic-sonnet
      provider: anthropic
      prompt_cache:
        enabled: true
        breakpoints: 4 # explicit cache_control markers
        ttl: '5m'

For self-hosted runtimes the equivalent is prefix caching, which saves compute rather than money.

The same constraint applies to both: the prefix must be stable. Order prompts stable-first — system instructions, then retrieved context, then the user turn.

Continuous batching

Batching is the main throughput lever on a self-hosted runtime, and the defaults assume traffic that may not resemble yours.

oculis-config.yaml

upstreams:
  - name: vllm-primary
    engine:
      max_num_seqs: 128 # concurrent sequences in flight
      max_num_batched_tokens: 8192 # per scheduler step
      scheduler_policy: fcfs # fcfs | priority
      preemption_mode: recompute # recompute | swap

Traffic shape	max_num_seqs	max_num_batched_tokens	Why
Interactive chat, short prompts	64–128	4096	Favor TTFT; many small sequences
RAG, long prompts, short answers	32–64	8192	Prefill-heavy; larger batches amortize
Batch/offline, throughput is everything	256+	16384	TTFT does not matter

Raising max_num_seqs without KV cache headroom backfires

More concurrent sequences need more KV cache. Past the cache ceiling the engine begins preempting and recomputing, and throughput falls — ERR_OCULIS_KV_CACHE_EXHAUSTED. Raise max_num_seqs and re-plan capacity together, never separately.

preemption_mode matters under pressure:

• recompute — discard the KV cache and redo prefill. Cheaper in memory, costly in compute.
• swap — move KV blocks to host memory. Cheaper in compute, needs PCIe bandwidth and pinned host RAM.

Use swap when PCIe is fast and prompts are long; recompute otherwise.

Speculative decoding

A small draft model proposes several tokens; the target model verifies them in one forward pass. When the draft is right, you get multiple tokens for roughly the cost of one step.

oculis-config.yaml

upstreams:
  - name: vllm-primary
    engine:
      speculative:
        enabled: true
        draft_model: meta-llama/Llama-3.2-1B-Instruct
        num_speculative_tokens: 5
        acceptance_threshold: 0.7

Gains are real but workload-dependent:

Workload	Typical acceptance	Throughput gain
Code completion	0.75–0.85	1.8–2.4×
Structured extraction (JSON)	0.70–0.80	1.6–2.1×
Conversational chat	0.55–0.65	1.2–1.5×
Creative writing	0.40–0.50	1.0–1.2×

curl -s http://localhost:9090/metrics | grep speculative

Expected output

oculis_speculative_acceptance_rate{upstream="vllm-primary"} 0.78
oculis_speculative_tokens_accepted_total{upstream="vllm-primary"} 41882094

Below ~0.5 acceptance, speculative decoding is a net loss

Rejected drafts are wasted compute plus the draft model’s VRAM. If acceptance sits below 0.5, disable it — you are paying for the draft model and getting nothing. Also note the draft and target must share a tokenizer, or you will see ERR_OCULIS_TOKENIZER_MISMATCH.

Response caching

For genuinely repeated requests — FAQ endpoints, deterministic classification — cache the whole response:

oculis:
  routes:
    - name: faq
      cache:
        enabled: true
        ttl_seconds: 3600
        key: [model, messages, temperature]
        max_entries: 100000
        # Only cache deterministic requests; caching temperature=0.9 is wrong.
        only_when:
          temperature_below: 0.01

Never cache across tenants

The cache key must include the tenant when responses contain tenant-scoped retrieved content, or you will serve one customer’s data to another. key: [model, messages, temperature] is safe only because it hashes the full message content — if you add a tenant-specific system prompt server-side, add tenant to the key explicitly.

Verification

Confirms throughput and cost moved in the intended direction and quality did not. Every change on this page is a trade — the verification is what tells you whether you got the better side of it.

1. Throughput against the baseline.

oculis bench run --route adaptive --profile mixed --duration 600s --compare baseline.json

Expected output

                        baseline      current     delta
  throughput            412 tok/s    987 tok/s   +139.6%
  requests/sec               8.2         19.4    +136.6%
  ttft p95              2,841ms      1,204ms     -57.6%
  cost per 1k req         $14.02        $4.74     -66.2%
  spec acceptance            n/a         0.78
  preemptions              8,412            0
  errors                       0            0

[PASS]  Throughput +139.6%, cost -66.2%, no error regression.

2. Quality did not regress. This is the check that makes the cost saving legitimate:

oculis eval run --route adaptive --suite eval/regression.jsonl --by-tier

Expected output

  tier        n      accuracy   vs_baseline   escalated
  simple    1,204      0.941        -0.006        2.1%
  standard    802      0.958        -0.002        0.8%
  complex     194      0.972        +0.001         n/a
  overall   2,200      0.949        -0.004        1.6%

[PASS]  Overall accuracy within tolerance (-0.4pp, threshold -2.0pp).

If the simple tier drops materially, tighten complexity_below so fewer requests land there.

3. Cost attribution reflects the change.

oculis cost report --since 7d --group-by route,tier

Edge cases and known limitations

• Semantic routing adds a classification hop (typically 8–20 ms). The cache makes repeated prompts free, but a fully unique traffic mix pays it every time.
• Larger batches raise TTFT. Throughput and latency trade against each other; you cannot maximize both. Decide which your users feel.
• Speculative decoding costs VRAM for the draft model. Re-plan capacity before enabling.
• Provider prompt-cache semantics differ. OpenAI caches automatically above a token threshold; Anthropic needs explicit breakpoints. Do not assume one provider’s behavior transfers.
• Response caching and streaming interact awkwardly. A cached response is returned in one chunk, so TTFT is excellent and the stream is not incremental. Some clients notice.
• Cost figures are estimates. They exclude provider discounts and batch pricing — see cost attribution.

Next steps

Benchmarking methodologyProduce the baseline every number on this page is measured against.

Capacity planningRe-plan VRAM after changing batching or adding a draft model.

Rate limits & token budgetsCap spend structurally as well as reducing it.