Rate Limits & Token Budgets

Applies tov1.4.0DifficultyIntermediateImpactNo restart

Two different problems share this page because they are enforced by the same machinery:

• Rate limits protect capacity. They stop one client from consuming the GPU tier.
• Budgets protect money. They stop a runaway loop from producing a five-figure invoice.

You almost certainly want both. Rate limits alone will not stop a slow, steady, expensive client.

Prerequisites

• Oculis Collector v1.4.0 or later
• Authentication configured — limits are enforced per tenant, and tenants come from API keys
• For multi-replica enforcement: a reachable Redis 6.2+ or Valkey 7+ instance
• Permissions: admin API key if applying at runtime rather than by restart

Verify all of the aboveoculis status --check-deps

Request rate limits

The first control: how many requests, how fast.

CLI / YAML

oculis-config.yaml

oculis:
  rate_limits:
    # Applies to any tenant without a more specific rule.
    default:
      requests_per_minute: 120
      burst: 30 # tokens in the bucket above the steady rate
      concurrent_requests: 16

    tenants:
      - tenant: acme
        requests_per_minute: 600
        burst: 150
        concurrent_requests: 64

      - tenant: internal-batch
        requests_per_minute: 3000
        burst: 500
        concurrent_requests: 256
        priority: low # yields to interactive traffic under contention

Python

limits.py

from oculis import RateLimit, TenantRateLimit

config.rate_limits = RateLimit(
    default=TenantRateLimit(
        requests_per_minute=120,
        burst=30,
        concurrent_requests=16,
    ),
    tenants=[
        TenantRateLimit(
            tenant="acme",
            requests_per_minute=600,
            burst=150,
            concurrent_requests=64,
        ),
        TenantRateLimit(
            tenant="internal-batch",
            requests_per_minute=3000,
            burst=500,
            concurrent_requests=256,
            priority="low",
        ),
    ],
)

cURL

curl -X PUT http://localhost:8080/admin/v1/rate-limits/acme \
  -H "Authorization: Bearer $OCULIS_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "requests_per_minute": 600,
    "burst": 150,
    "concurrent_requests": 64
  }'

requests_per_minute is a token bucket, not a fixed window. burst is how far above the steady rate a client may spike before being throttled — set it to roughly 25% of the per-minute rate so normal bursty traffic is not punished.

Exceeding a limit returns ERR_OCULIS_RATE_LIMIT_LOCAL with a Retry-After header.

concurrent_requests is the one that actually protects a GPU

Requests per minute is a poor proxy for GPU load, because a 32K-token request occupies a node far longer than a 200-token one. concurrent_requests caps how many of a tenant’s requests can occupy the tier simultaneously, which is much closer to the thing you are protecting.

Token and spend budgets

The second control: how much consumption, over what period.

CLI / YAML

oculis-config.yaml

oculis:
  budgets:
    - tenant: acme
      tokens_per_day: 5_000_000
      spend_per_month_usd: 2000
      on_exceeded: downgrade # or `reject`
      downgrade_route: cheap-tier
      alert_thresholds: [0.5, 0.8, 0.95]

    - tenant: trial
      tokens_per_day: 100_000
      spend_per_month_usd: 25
      on_exceeded: reject

Python

limits.py

from oculis import Budget

config.budgets = [
    Budget(
        tenant="acme",
        tokens_per_day=5_000_000,
        spend_per_month_usd=2000,
        on_exceeded="downgrade",
        downgrade_route="cheap-tier",
        alert_thresholds=[0.5, 0.8, 0.95],
    ),
    Budget(
        tenant="trial",
        tokens_per_day=100_000,
        spend_per_month_usd=25,
        on_exceeded="reject",
    ),
]

reject vs. downgrade

Behavior	What the client sees	Use when
reject	402 ERR_OCULIS_BUDGET_EXCEEDED	A hard cap is the point — trials, untrusted tenants.
downgrade	A response from a cheaper model	Continuity matters more than consistency — paying customers.

downgrade is usually the right default for production tenants: the tenant keeps working in a degraded mode instead of experiencing an outage on the last day of the month.

Spend tracking is an estimate until the invoice arrives

Costs are computed from token counts and a per-model price table. Provider-side discounts, prompt-caching credits, and batch pricing are not reflected. Treat spend_per_month_usd as a guardrail, not as accounting. If your counts drift from provider billing, that is ERR_OCULIS_TOKENIZER_MISMATCH.

Price table

Downgrade and spend enforcement need to know what things cost:

oculis-config.yaml

oculis:
  pricing:
    # USD per million tokens
    - model: gpt-4o-mini
      input: 0.15
      output: 0.60
    - model: meta-llama/Llama-3.1-70B-Instruct
      input: 0.0 # self-hosted; amortize separately
      output: 0.0
      infra_cost_per_hour_usd: 9.80 # optional, for cost attribution

Distributed enforcement

By default every replica keeps its own counters. Two replicas with a 600 rpm limit will admit 1200 rpm in aggregate — which is usually not what you meant.

CLI / YAML

oculis-config.yaml

oculis:
  rate_limits:
    backend: redis
    redis:
      url: ${env:OCULIS_REDIS_URL} # redis://:pass@host:6379/0
      key_prefix: 'oculis:rl:'
      timeout_ms: 50
      # If Redis is unreachable, fall back to per-replica local counters
      # rather than failing requests.
      fail_open: true

Helm

values.yaml

rateLimits:
  backend: redis
  redis:
    urlSecretRef:
      name: oculis-redis
      key: url
    keyPrefix: 'oculis:rl:'
    timeoutMs: 50
    failOpen: true

fail_open is the right default here

A Redis outage should degrade enforcement precision, not take down inference. With fail_open: true each replica falls back to local counters — limits become approximate but traffic keeps flowing. Set it to false only when over-admitting is worse than an outage.

Response headers

Every response carries the current state so clients can self-regulate rather than discover limits by hitting them:

X-Oculis-RateLimit-Limit: 600
X-Oculis-RateLimit-Remaining: 412
X-Oculis-RateLimit-Reset: 27
X-Oculis-Budget-Tokens-Remaining: 3891204
X-Oculis-Budget-Spend-Remaining-Usd: 431.18
Retry-After: 27

Client libraries should read Retry-After and back off with jitter. See rate limit cascades for what happens when they do not.

Apply and verify

1. Validate and apply.

   oculis config validate --file oculis-config.yaml --explain
   oculis apply --file oculis-config.yaml --reload

   Rate limits and budgets apply on --reload without a restart.

2. Confirm the effective limit for a tenant.

   oculis limits show --tenant acme

Verification

Confirms the limit is actually enforced, the correct status is returned, and budget counters are being tracked. A limit you have not tripped on purpose is not a limit you can rely on.

1. Trip the rate limit deliberately.

oculis limits test --tenant acme --rate 900 --duration 10s

Expected output

[INFO]  Target 900 rpm against limit 600 rpm (burst 150)
[INFO]  Sent 150 requests in 1.0s — all admitted (burst absorbed)
[INFO]  Steady state: 600 admitted/min, 300 rejected/min
[INFO]  Rejections returned 429 with Retry-After (min 1s, max 12s)
[PASS]  Rate limit enforced at 600 rpm (±2%).

2. Confirm the counters are exported.

curl -s http://localhost:9090/metrics | grep -E 'oculis_(ratelimit|budget)_'

Expected output

oculis_ratelimit_rejected_total{tenant="acme",reason="rpm"} 50
oculis_ratelimit_admitted_total{tenant="acme"} 100
oculis_budget_tokens_used{tenant="acme",window="day"} 1108796
oculis_budget_spend_usd{tenant="acme",window="month"} 1568.82

3. Confirm distributed enforcement, if configured. Run the same test against two replicas and confirm the aggregate stays at 600 rpm rather than 1200:

oculis limits test --tenant acme --rate 1500 --duration 10s --via-load-balancer

Edge cases and known limitations

• Streaming requests count once, at admission. A stream that runs for two minutes occupies a concurrency slot for that whole time but consumes one request from the rate bucket.
• Token budgets are checked before generation, using an estimate. Actual output tokens are reconciled after the response, so a tenant can overshoot a daily cap by roughly one request.
• Budget windows are UTC. tokens_per_day resets at 00:00 UTC, not in the tenant’s timezone.
• Priority is advisory. priority: low yields under contention but does not guarantee starvation avoidance for high-priority traffic during sustained overload.
• A downgrade route must exist and be reachable. If downgrade_route names a route that is unhealthy, budget-exceeded traffic fails rather than downgrading.

Next steps

Authentication & RBACTenants come from API keys — configure them first if you have not.

Telemetry & observabilityAlert on budget burn rate before the cap is hit.

Throughput & costReduce spend structurally instead of just capping it.