RAG Pipeline from Scratch

The problem

Most RAG tutorials start and end with langchain.from_documents() — a few lines that hide every interesting decision inside a framework call. I wanted to understand what's actually happening: how text gets chunked, how embeddings turn meaning into numbers, how similarity search actually finds relevant context, and how that context turns into a grounded answer. So I built the whole pipeline by hand, running entirely locally against Ollama with no cloud APIs and no LangChain.

Starting point: RAG in its simplest form

The first version was intentionally minimal: load one PDF, split it into overlapping character-based chunks, embed each chunk with Ollama's nomic-embed-text model, and store the embeddings in a plain Python list. For a question, I embedded it the same way and computed cosine similarity by hand against every stored chunk, took the top matches, and passed them to llama3.2 as context for the final answer. No vector database, no index — just loops and NumPy. It worked, and more importantly, it made every step visible.

Scaling to multiple documents

A single PDF doesn't test much. I extended the pipeline to load an entire folder of PDFs and tag every chunk with a source field so retrieval could later be scoped to a specific document. This is also where the manual cosine-similarity loop started to show its limits — it's fine for ten chunks, unworkable for hundreds. I replaced it with ChromaDB, a persistent local vector store: it indexes embeddings for fast nearest-neighbor search and survives between runs, so I stopped re-embedding the same documents every time I restarted the notebook.

Comparison-aware generation

Once the pipeline handled multiple documents, a natural use case appeared: comparing two versions of the same document (I tested this against two versions of a course's exam guidelines — "what changed between the old and new AST exam rules?"). A single similarity search across a mixed collection doesn't answer that well, since it just returns whichever chunks are closest regardless of which document they're from. I added a filtered retrieval path using Chroma's where clause to pull top matches from each document separately, then built a comparison-specific prompt that explicitly labels an OLD and a NEW context block and asks the model to identify what changed between them.

Automatic query routing

Manually deciding "is this a lookup or a comparison question?" and passing the right filename defeats the point of a natural-language interface. So I added a routing layer: the question is first sent to llama3.2 with a classification prompt that returns structured JSON — either {"type": "lookup"} or {"type": "comparison", "item_a": ..., "item_b": ...}. For comparisons, the extracted phrases (e.g. "old AST exam") aren't real filenames, so I match them to actual filenames by embedding both the phrase and every candidate filename and picking the closest one by cosine similarity — a second, smaller RAG problem nested inside the first. This surfaced a real failure mode worth designing around: the match always returns something, even with no good match, so a low-confidence guess can silently produce a wrong comparison. I added an explicit guard so an unmatched document fails loudly with a clear message instead of crashing several layers deeper inside Chroma's filter validation — which is exactly what happened once, mid-project: the vector database was cleared and rebuilt but never reloaded before the next query, and the resulting None filename got passed straight into Chroma's where filter, producing an opaque ValueError several stack frames removed from the actual cause.

Sentence-aware chunking

The original chunker split text at fixed character offsets, which regularly cut sentences in half mid-thought. I replaced it with a sentence-aware chunker built on NLTK's tokenizer: it groups whole sentences up to a target length, with a small sentence-level overlap between consecutive chunks, so retrieval never hands the model a fragment that starts or ends mid-sentence.

From notebook to a real tool

The full system — loading, chunking, embedding, storage, retrieval, comparison, and routing — still only lived inside notebook cells, which meant testing a new question meant editing a variable and re-running cells by hand. Not something you could hand to someone else. I extracted all of that logic into a standalone Streamlit app (app.py): a text box, a submit button, and a loading spinner while the local LLM thinks, all running against the already-persisted Chroma collection built earlier — no re-embedding needed to use it. streamlit run app.py now opens a real browser-based tool where any question, lookup or comparison, gets routed and answered automatically.

Outcome

What started as "let's see how RAG actually works under the hood" became a locally-run, multi-document Q&A tool with automatic query classification, document comparison, and a real UI — built without a single line of LangChain, and without sending a single request to an external API. The bigger takeaway was less about the specific pipeline and more about the debugging instinct it built: most of the hardest bugs weren't in the "smart" parts (the LLM prompts) — they were in the plumbing between them (an unrefreshed vector store, an unguarded None), which is exactly the kind of thing a framework would have hidden from me.