Knowledge Base / RAG

What it is

The Knowledge Base is TurfAI's RAG layer. Enable indexing on a document and you can search it by meaning, not just keywords. Natural-language queries return top-K passages with similarity scores and an AI-synthesized answer carrying source citations (document_title, chunk_text, similarity_score, and a signed_url to the source).

For the underlying idea, see the RAG concept. This page covers the module.

When to use it

Use it whenever AI needs to answer from your documents — policy Q&A, support knowledge bases, contract lookups — or when an agent needs a search_documents tool. It also powers the Chatbot.

How it works

Process flow — index then query:

Call flow — a RAG query:

Sub-features

• Document indexing — async with status on the document's rag_processing_status (not_started → queued → processing → completed, or failed), force_reprocess, chunk count (rag_chunk_count), embedding-model metadata.
• Semantic search — top-K with similarity scoring, optional reranking, configurable similarity threshold.
• Conversation sessions — persistent multi-turn chat with session ids, history, deletion, titles.
• Collections — group documents into named collections and scope queries to them.
• Two backends — Google File Search (default) and pgvector (legacy), selected per deployment.

Configuration parameters

RAG config has two layers: deployment/indexing config (ops sets it through env vars, and it applies to all queries) and per-query config (you pass it on each /api/rag/query call).

Indexing & backend (deployment level)

Per-query parameters (POST /api/rag/query)

Tips & best practices

• Use a collection when you have a distinct knowledge domain (one product, one customer, one policy set). Scope queries with filters.collection_ids so unrelated documents can't leak into answers.
• Document hygiene: prefer machine-readable PDFs, DOCX, TXT, or Markdown; keep one topic per document; re-run with force_reprocess=true after meaningful edits so the chunk index matches the live file.
• Choosing a backend: start with Google File Search (API key only, embeddings handled by Gemini, ~90% cheaper, auto-scaling). Move to pgvector only when you need on-prem/full control of the vector store.
• Tuning similarity_threshold: raise it (e.g. 0.5–0.6) when answers pull in loosely related passages. Lower it (e.g. 0.2–0.3) when the search misses relevant documents — especially for multi-lingual content. Turn on use_reranking before raising the threshold; reranking usually fixes "right docs, wrong order" without dropping recall.
• Citations: every answer includes sources[] with document_title, chunk_text, similarity_score, page_number, and a 1-hour signed_url. Show these in the UI so users can verify the answer. If sources is empty, treat the answer as ungrounded.

Concrete examples

1. Enable RAG, then poll status

# Enable indexing on document 42
POST /api/documents/42/enable-rag
{ "force_reprocess": false }
# → { "status": "queued", "job_id": "rag-embed-42-1718800000000", "document_id": "42" }

# Poll until completed
GET /api/documents/42/rag-status
# → { "processing_status": "queued",     "chunk_count": 0 }   # then...
# → { "processing_status": "processing", "chunk_count": 0 }   # then...
# → { "processing_status": "completed",  "chunk_count": 37 }
# status values: not_started | queued | processing | completed | failed
# (the endpoint returns the document's rag_processing_status as processing_status)

2. Run a RAG query

POST /api/rag/query
{
  "query": "What is the notice period for terminating the MSA?",
  "top_k": 5,
  "similarity_threshold": 0.3,
  "use_reranking": true,
  "filters": { "collection_ids": [7] }
}

Response (shape from QueryResponse / Source):

{
  "answer": "The MSA requires 60 days' written notice to terminate for convenience [1].",
  "sources": [
    {
      "document_id": 42,
      "document_title": "Master Services Agreement.pdf",
      "chunk_text": "...either party may terminate this Agreement for convenience upon sixty (60) days' prior written notice...",
      "chunk_index": 12,
      "similarity_score": 0.71,
      "page_number": 8,
      "file_url": "gs://turfai-docs/42/msa.pdf",
      "signed_url": "https://storage.googleapis.com/turfai-docs/42/msa.pdf?X-Goog-Expires=3600..."
    }
  ],
  "confidence": 0.82,
  "session_id": "sess_a1b2c3",
  "processing_time_ms": 940
}

3. Multi-turn session — reuse the returned session_id on the next call to keep context:

POST /api/rag/query
{ "query": "And what about termination for cause?", "session_id": "sess_a1b2c3" }

The service resolves "what about..." against the prior turn. If you omit session_id, the service creates a new session automatically. List, inspect, or delete sessions through /api/rag/sessions.

Limitations

• Standalone images (JPEG/PNG) may not be supported by Google File Search — convert to PDF first.
• Google File Search auto-OCR of scanned PDFs is not explicitly documented by Google. Gemini's native vision usually reads them, but test before relying on it; in-boundary OCR for scanned PDFs is coming soon (today's fallback is TurfAI OCR → pgvector).
• Per-document file limits (File Search): up to 100 MB / 1000 pages; pages scaled to 3072×3072.
• Data Shield does not yet tokenise the RAG chat path (coming in v3.5) — do not send untokenised PII through RAG chat expecting masking.
• Persistent / cross-call session scope beyond the current store is coming soon.

Troubleshooting

How to test

Run this end-to-end happy path:

1. Upload a machine-readable PDF (DOCX/TXT/MD also work).
2. Enable RAG: POST /api/documents/:id/enable-rag; expect status: queued.
3. Wait for completed: poll GET /api/documents/:id/rag-status until processing_status is completed and chunk_count > 0. If it lands on failed, read rag_error.
4. Query: POST /api/rag/query with a question whose answer is in the document.
5. Confirm a cited answer: the response has a non-empty answer and sources[] with the right document_title, a page_number, a similarity_score above your threshold, and a working signed_url.
6. Multi-turn: send a follow-up question with the returned session_id and confirm the answer respects the earlier turn's context.
7. Collection scope (optional): add the document to a collection and confirm filters.collection_ids returns it while excluding others.

Cross-links & roadmap

• Chatbot — public/embedded chat surface built on this layer.
• RAG concept — chunking, embeddings, and retrieval explained.
• Data Shield — note: RAG chat is not tokenised by Data Shield until v3.5.
• Coming soon: in-boundary OCR for scanned PDFs; persistent (cross-call) session scope.

Dependencies

• RAG query service (Python) — embeddings, vector search, conversation memory, citations.
• RAG embeddings worker — chunks + embeds documents off a Redis queue.
• Vector store — Google File Search or PostgreSQL + pgvector.
• LLM / embedding service — Gemini via Vertex AI (File Search) or TurfAI LLM service (pgvector).
• Document management + GCS — source documents and 1-hour signed URLs.

APIs used

See the synced feature reference for the full RAG surface.