Building a RAG Pipeline for Oracle Documentation with OCI Generative AI and FAISS

When I experimented with using an LLM as an Oracle technical assistant, I did not want the model to answer purely from its training data. Oracle questions frequently depend on exact syntax, release-specific behavior, and details buried deep inside documentation. That makes this a good fit for retrieval-augmented generation (RAG).

The prototype I worked with used FAISS as the vector index, a sentence-transformer embedding model for local retrieval, a cross-encoder reranker to improve the ordering of retrieved passages, and OCI Generative AI as the final language-model layer.

1. The Pipeline I Wanted

Oracle documentation / technical notes
              ↓
      clean + split into chunks
              ↓
         text embeddings
              ↓
          FAISS index
              ↓
 question → vector similarity search
              ↓
      top candidate passages
              ↓
   cross-encoder reranking
              ↓
      best 3–5 passages
              ↓
      OCI Generative AI
              ↓
 grounded technical answer

The important design choice is that retrieval and generation are separate. I can inspect what FAISS retrieved, inspect what the reranker promoted or demoted, and only then look at the model's answer. That separation makes debugging much easier.

2. Represent Every Chunk with Metadata

from dataclasses import dataclass

@dataclass
class DocChunk:
    text: str
    source: str
    section: str
    chunk_id: int

A simple chunking function can retain source and section information while creating overlapping windows:

def split_text(text, source, section,
               chunk_size=1000, overlap=180):
    chunks = []
    start = 0
    chunk_id = 0

    while start < len(text):
        end = min(start + chunk_size, len(text))
        part = text[start:end].strip()

        if part:
            chunks.append(
                DocChunk(
                    text=part,
                    source=source,
                    section=section,
                    chunk_id=chunk_id,
                )
            )
            chunk_id += 1

        if end == len(text):
            break

        start = end - overlap

    return chunks

For technical documentation I prefer semantic boundaries—headings, paragraphs, and code examples—over blindly cutting every N characters. The overlap reduces the chance that an explanation is split exactly where the important context begins.

3. Build a FAISS Index

import faiss
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer(
    "sentence-transformers/all-MiniLM-L6-v2"
)

texts = [chunk.text for chunk in chunks]

vectors = embedder.encode(
    texts,
    normalize_embeddings=True,
    convert_to_numpy=True,
).astype("float32")

index = faiss.IndexFlatIP(vectors.shape[1])
index.add(vectors)

faiss.write_index(index, "oracle_docs.faiss")

Because the embeddings are normalized, inner product is equivalent to cosine similarity. I persist the FAISS index together with the chunk metadata so every vector position still maps to the correct source passage.

4. Retrieve More Candidates Than I Ultimately Need

def retrieve_candidates(question, top_k=12):
    q = embedder.encode(
        [question],
        normalize_embeddings=True,
        convert_to_numpy=True,
    ).astype("float32")

    scores, ids = index.search(q, top_k)

    results = []
    for score, idx in zip(scores[0], ids[0]):
        if idx == -1:
            continue

        results.append({
            "score": float(score),
            "chunk": chunks[idx],
        })

    return results

This first stage is optimized for recall: find passages that may be relevant, even if their ordering is imperfect.

5. Rerank with a Cross-Encoder

Vector similarity is fast, but the nearest vector is not always the passage that best answers the question. I used a cross-encoder based on cross-encoder/ms-marco-MiniLM-L-6-v2 to rerank the candidate set.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder(
    "cross-encoder/ms-marco-MiniLM-L-6-v2"
)

def rerank(question, candidates, keep=4):
    pairs = [
        (question, item["chunk"].text)
        for item in candidates
    ]

    scores = reranker.predict(pairs)

    ranked = sorted(
        zip(scores, candidates),
        key=lambda x: x[0],
        reverse=True,
    )

    return [
        {
            "rerank_score": float(score),
            **item,
        }
        for score, item in ranked[:keep]
    ]

The FAISS stage compares vectors. The cross-encoder evaluates the question and passage together. That extra computation is worthwhile when reducing a dozen candidates to the few passages that deserve context-window space.

6. Build a Traceable Context

def build_context(results):
    sections = []

    for i, item in enumerate(results, start=1):
        c = item["chunk"]
        sections.append(
            f"[Source {i}] {c.source} / {c.section}\\n"
            f"{c.text}"
        )

    return "\\n\\n".join(sections)

I then instruct the model to stay grounded in the retrieved evidence:

prompt = f"""
You are an Oracle technical assistant.

Answer the question using the supplied documentation context.
If the context is insufficient, say so instead of inventing
Oracle syntax, parameters, or behavior.

CONTEXT
-------
{context}

QUESTION
--------
{question}
"""

7. Send the Grounded Request to OCI Generative AI

import oci
from oci.generative_ai_inference import GenerativeAiInferenceClient
from oci.generative_ai_inference.models import (
    ChatDetails,
    GenericChatRequest,
    OnDemandServingMode,
    UserMessage,
    TextContent,
)

config = oci.config.from_file("~/.oci/config", "DEFAULT")
client = GenerativeAiInferenceClient(config)

def ask_oci(prompt, compartment_id, model_id):
    request = GenericChatRequest(
        messages=[
            UserMessage(
                content=[TextContent(text=prompt)]
            )
        ],
        temperature=0.1,
        max_tokens=1200,
    )

    details = ChatDetails(
        compartment_id=compartment_id,
        serving_mode=OnDemandServingMode(
            model_id=model_id
        ),
        chat_request=request,
    )

    response = client.chat(details)

    message = (
        response.data.chat_response
        .choices[0]
        .message
    )

    return message.content[0].text

For technical Q&A I keep the temperature low. I want the model to explain the evidence clearly, not creatively improvise around it.

8. Make Retrieval Observable

for item in reranked:
    c = item["chunk"]

    print(
        c.source,
        c.section,
        item["score"],
        item["rerank_score"],
    )

When an answer is weak, I can distinguish four very different failures:

• Retrieval failure: FAISS never found the correct passage.
• Ranking failure: the correct passage was retrieved but ranked too low.
• Chunking failure: important context was separated from the passage.
• Generation failure: correct evidence reached the model but was interpreted poorly.