Embeddings and Local RAG

Retrieval-augmented generation, fully offline

A small local model does not know your college's rulebook or your company wiki, and its context window cannot hold all of it. Retrieval-augmented generation (RAG) solves this in two stages. Offline, you split documents into chunks (a few hundred tokens each, often with some overlap) and convert each chunk into an embedding: a vector of numbers from an embedding model, where texts with similar meaning land close together. At question time, you embed the question, find the most similar chunks (commonly by cosine similarity), and paste them into the prompt with an instruction to answer only from that context. With Ollama, embed on a model such as nomic-embed-text produces vectors, and a chat model writes the answer, so no data leaves the machine. Quality depends mostly on retrieval: bad chunking or a weak embedding model means the right passage never reaches the LLM.

Ranking chunks by cosine similarity

Toy vectors to show the maths; the next snippet gets real ones from Ollama.

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

# Toy 3-d vectors; real embedding models return hundreds of dimensions
docs = {
    "refund policy": [0.9, 0.1, 0.0],
    "hostel timings": [0.1, 0.8, 0.3],
    "exam schedule": [0.0, 0.3, 0.9],
}
query = [0.8, 0.2, 0.1]  # "how do I get my money back?"
ranked = sorted(docs, key=lambda d: cosine(query, docs[d]), reverse=True)
for d in ranked:
    print(f"{cosine(query, docs[d]):.3f}  {d}")

Output:

0.984  refund policy
0.378  hostel timings
0.190  exam schedule

Real embeddings and a grounded answer

Needs a running server with nomic-embed-text and llama3.2 pulled.

import ollama

chunks = [
    "Fee refunds are processed within 30 days of withdrawal.",
    "The hostel gate closes at 10 pm on weekdays.",
]
vecs = ollama.embed(model="nomic-embed-text", input=chunks).embeddings

question = "How long does a fee refund take?"
q = ollama.embed(model="nomic-embed-text", input=question).embeddings[0]

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (sum(x * x for x in a) ** 0.5 * sum(y * y for y in b) ** 0.5)

best = max(range(len(chunks)), key=lambda i: cosine(q, vecs[i]))
prompt = f"Answer only from this context:\n{chunks[best]}\n\nQuestion: {question}"
print(ollama.chat(model="llama3.2", messages=[{"role": "user", "content": prompt}]).message.content)

The local RAG pipeline

Indexing happens once; the last three stages run for every question. The stages, in order: chunk documents -> embed chunks into vectors -> retrieve top matches for the question -> build a grounded prompt -> local chat model answers.

Quick check: Your local RAG bot gives confident but wrong answers. Which check should usually come first?

• Raise the temperature
• Switch to a larger quantization of the chat model immediately
• Remove the system prompt
• Print the retrieved chunks to see whether the right passage was found