RAG & Vector Databases

Applies tov1.4.0DifficultyIntermediateImpactRolling restart

Oculis does not store your vectors. It connects to a store you operate, retrieves context at request time, and assembles it into the prompt under a token budget.

Two decisions on this page cause most production RAG problems: how you bound retrieved context, and what you do when the embedding model changes.

Prerequisites

• Oculis Collector v1.4.0 or later
• A reachable vector store
• An embedding model, either via a provider API or a local runtime
• An existing collection with a known dimension and distance metric
• Network: egress to the store (Qdrant 6333, Milvus 19530, Postgres 5432) and to the embedding provider on 443
• Permissions: read access to the collection; write access only if Oculis manages ingestion

Verify all of the aboveoculis status --check-deps

Step 1 — Connect the store

Qdrant

oculis-config.yaml

oculis:
  vector_stores:
    - name: docs
      provider: qdrant
      endpoint: 'http://qdrant.data.svc.cluster.local:6333'
      api_key: ${env:QDRANT_API_KEY}
      collection: product_docs
      timeout_ms: 800
      metric: cosine

Milvus

oculis-config.yaml

oculis:
  vector_stores:
    - name: docs
      provider: milvus
      endpoint: 'milvus.data.svc.cluster.local:19530'
      username: ${env:MILVUS_USER}
      password: ${env:MILVUS_PASSWORD}
      collection: product_docs
      partition: v2
      timeout_ms: 800
      metric: ip

pgvector

oculis-config.yaml

oculis:
  vector_stores:
    - name: docs
      provider: pgvector
      dsn: ${env:POSTGRES_DSN}
      table: product_docs
      embedding_column: embedding
      content_column: chunk_text
      metadata_column: metadata
      timeout_ms: 800
      metric: cosine
      pool:
        max_connections: 10

Pinecone

oculis-config.yaml

oculis:
  vector_stores:
    - name: docs
      provider: pinecone
      api_key: ${env:PINECONE_API_KEY}
      index: product-docs
      namespace: production
      timeout_ms: 800

Set a tight timeout

Retrieval is on the critical path for time to first token. An 800 ms timeout that occasionally fails open is better than a 10 s timeout that makes every slow query a user-visible stall. Pair it with fail_open below.

Step 2 — Configure the embedding model

oculis-config.yaml

oculis:
  embeddings:
    - name: default-embed
      provider: openai
      model: text-embedding-3-large
      version: '2024-10-01' # pin it — see below
      dimensions: 3072
      batch_size: 96
      api_key: ${env:OPENAI_API_KEY}

Pinning embedding versions

Providers update floating model aliases without notice. When that happens, new queries are embedded by a different model than the corpus was, and retrieval quality degrades slowly with no configuration change on your side. That is ERR_OCULIS_EMBEDDING_MODEL_DRIFT.

Always pin version, and treat a version bump as a re-embed project rather than a config tweak:

embeddings:
  - name: default-embed
    provider: openai
    model: text-embedding-3-large
    version: '2024-10-01' # never omit this in production
    on_version_drift: error # error | warn | ignore

Step 3 — Configure retrieval

oculis-config.yaml

oculis:
  retrieval:
    - name: docs-retrieval
      store: docs
      embedding: default-embed

      top_k: 8
      score_threshold: 0.35 # drop weak matches entirely

      # Bound context by TOKENS, not by document count.
      max_context_tokens: 3000
      overflow_strategy: truncate_lowest_score

      hybrid:
        enabled: true
        alpha: 0.7 # 1.0 = pure vector, 0.0 = pure keyword

      rerank:
        enabled: true
        model: bge-reranker-v2-m3
        top_n: 4 # rerank 8 candidates down to 4

      filters:
        - field: tenant
          value: '${request.tenant}' # never leak across tenants

      fail_open: true # answer without grounding rather than erroring

Always filter by tenant

A vector store shared across tenants without a tenant filter will return one customer’s documents to another. ${request.tenant} resolves from the authenticated key — which is another reason to configure authentication before retrieval.

Why a token budget rather than top_k alone

top_k: 8 bounds document count, not size. Eight long chunks can exceed the model window on their own, which produces ERR_OCULIS_CONTEXT_OVERFLOW only for the requests that retrieved large documents — an intermittent failure that is miserable to diagnose.

max_context_tokens makes the bound explicit. overflow_strategy decides what gives:

Strategy	Behavior
truncate_lowest_score	Drop the weakest matches until it fits. Recommended.
truncate_tail	Trim the end of each chunk. Preserves breadth, loses depth.
error	Fail the request. Use when silent context loss is worse.

Connection health and failure modes

retrieval:
  - name: docs-retrieval
    fail_open: true
    fail_open_message: 'Answering without retrieved context.'
    circuit_breaker:
      failure_threshold: 5
      open_duration_seconds: 30

fail_open	Store unreachable	Choose it when
true	Answer from model knowledge alone	A degraded answer beats no answer.
false	ERR_OCULIS_VECTOR_STORE_UNREACHABLE	An ungrounded answer would be dangerous or misleading.

fail_open produces confident, ungrounded answers

With fail_open: true, a store outage does not look like an outage — it looks like the assistant quietly losing its citations while remaining fluent. Alert on oculis_retrieval_fail_open_total, and consider surfacing a UI hint when grounding is unavailable. Silence is the failure mode here.

Changing embedding models

You cannot change embedding models in place. Dimensions differ, and even at equal dimensions the vector spaces are not comparable — a mismatch surfaces as ERR_OCULIS_VECTOR_DIM_MISMATCH.

Use an alias swap so the cutover is atomic and reversible:

1. Create a new collection with the new model’s dimension. Do not touch the live one.

   oculis vector create-collection \
     --store docs \
     --name product_docs_v3 \
     --dimensions 3072 \
     --metric cosine

2. Re-embed the corpus into it. This is the slow part; it does not affect live traffic.

   oculis vector reindex \
     --store docs \
     --source product_docs_v2 \
     --target product_docs_v3 \
     --embedding default-embed \
     --batch-size 96 \
     --concurrency 4

3. Evaluate retrieval quality against a held-out query set before cutting over.

   oculis vector evaluate \
     --store docs \
     --collection product_docs_v3 \
     --queries eval/queries.jsonl \
     --metrics recall@4,mrr

   Example output

   collection            recall@4    mrr      queries
   product_docs_v2         0.812    0.694        500
   product_docs_v3         0.867    0.741        500
   [PASS] New collection improves recall@4 by 5.5pp.

4. Swap the alias. This is the only step that touches production, and it is instant.

   oculis vector alias set --store docs --alias product_docs --target product_docs_v3

5. Keep the old collection for at least one deploy cycle so rollback is an alias swap rather than a re-embed.

Verification

Confirms the store is reachable, dimensions agree, retrieval returns sensible results, and tenant filtering actually isolates. The last check is a security control, not a quality check.

1. Connection and dimension agreement.

oculis vector test --store docs

Expected output

[INFO]  Store "docs"  provider=qdrant  endpoint=http://qdrant.data.svc:6333
[INFO]  Connection    OK  (latency 6ms)
[INFO]  Collection    product_docs  vectors=1,284,902  dim=3072  metric=cosine
[INFO]  Embedding     text-embedding-3-large@2024-10-01  dim=3072
[PASS]  Dimensions match (3072 == 3072).

2. A real query returns scored, relevant results.

oculis retrieval test \
  --name docs-retrieval \
  --query "How do I configure gateway failover?" \
  --show-scores

Expected output

[INFO]  Embedded query in 61ms; searched 1,284,902 vectors in 11ms
  0.871  configuration/fallback.mdx#step-2-choose-a-routing-strategy
  0.844  configuration/fallback.mdx#step-3-configure-the-circuit-breaker
  0.792  configuration/gateway.mdx#choosing-a-strategy
  0.641  troubleshooting/inference-streaming.mdx#gateway-timeouts
[INFO]  4 chunks after rerank, 1,912 tokens (budget 3,000)
[PASS]  Retrieval returned 4 results above threshold 0.35.

3. Tenant isolation holds.

oculis retrieval test --name docs-retrieval \
  --query "quarterly revenue" --as-tenant acme --assert-filter tenant=acme

Expected output

[INFO]  Applied filter: tenant = "acme"
[INFO]  8 candidates, 8 matched filter, 0 leaked
[PASS]  Tenant isolation verified.

A non-zero leaked count is a data-exposure bug. Stop and fix the filter before proceeding.

Edge cases and known limitations

• Reranking adds latency on the critical path. A cross-encoder reranker typically costs 40–120 ms. Worth it for quality; measure it against your TTFT budget.
• score_threshold is metric-dependent. A cosine threshold of 0.35 means something different under inner product or L2. Re-tune it whenever you change metric.
• Hybrid search requires a keyword index. Enabling hybrid against a store without one silently degrades to pure vector search.
• Oculis does not chunk or ingest by default. It queries what you have indexed. Ingestion pipelines are yours to run, which means chunk size and overlap are decided outside this config.
• Metadata filters are applied by the store, not by Oculis. A store that does not support the filter will ignore it rather than error — verify isolation with check 3 above rather than assuming.

Next steps

Guardrails & safetyRetrieved content is untrusted input. Fence it.

Context window overflowDiagnosing requests that fail only when retrieval returns a lot.

Latency tuningPrefix caching interacts strongly with retrieved context.