RagNest: Multi-Tenant RAG, Versioned Pipeline Registry

The Problem

This is the direct next step from RAG Pipeline from Scratch, an earlier project built to understand what a RAG framework actually hides: manual chunking, hand-rolled cosine similarity, a local ChromaDB store, all wired together in a notebook and then a single-user Streamlit app. That project answered "how does retrieval-augmented generation actually work." It didn't answer, and was never meant to: what happens when more than one person needs to use it, when the model or retrieval strategy needs to change without a redeploy, or when you need to know which configuration actually produced a given answer months later. RagNest is that next question, built from the ground up rather than extended in place, because the answer changes the schema, not just the feature set.

The first version of RagNest still worked the way most RAG side projects do: one LLM, one embedding model, one chunking strategy, all pinned in environment variables. That's fine for a demo. It falls apart the moment a second team needs a different model, or you want to try a better embedding model without editing .env and restarting the process, or you want to know which model actually generated a given answer several config changes ago. I wanted the version of this a real product would need: multiple workspaces, each with their own documents and conversations, sharing a platform-wide catalog of models and retrieval strategies that admins manage at runtime — with every past answer still traceable to the exact configuration that produced it, and with access control that couldn't be talked out of itself by an admin's own mistake.

System Overview

FastAPI over async SQLAlchemy/Alembic on Postgres, with Weaviate for vectors, S3 (LocalStack locally) for file storage, SQS for event-driven ingestion, and Ollama for local inference — the whole stack runs via one docker compose up, self-bootstrapping its own migrations, bucket, queue, and vector schema. Document ingestion is event-driven end to end: a client asks for a presigned S3 upload URL, uploads directly to S3, and an S3-event-triggered SQS message hands the file to a worker that extracts text, chunks it, embeds it, and writes vectors to Weaviate — all off the request path, so a large PDF never blocks an HTTP response. Chat answers stream over SSE, generated by a retrieval pipeline that can run dense-vector search, BM25, or a weighted hybrid of both, narrowed by an optional cross-encoder reranking pass, with query rewriting, context compression, and multi-query retrieval available as configurable steps rather than fixed behavior. A Next.js/TypeScript admin UI sits on top for managing all of it, but the part actually worth walking through is everything underneath: how the configuration those choices come from is modeled, and how access to change any of it is controlled.

The Architecture Rewrite: From a Flat Settings Row to a Versioned Pipeline Registry

The platform originally stored its entire configuration in one rag_runtime_settings row — 25-plus columns, mutated in place every time an admin changed a setting. It worked, but it couldn't answer questions a real product needs answered: which exact configuration produced a given document's vectors, whether a bad change could be rolled back, or how two different assistants could eventually run two different pipelines without duplicating half the schema. Getting the replacement right took three rounds of architecture review that were each rejected on design intent, not code — no table got created until the shape itself was agreed on.

The first rejected shape was a strict containment tree: Workspace ⊃ Assistant ⊃ Pipeline. It looks natural until a KnowledgeBase or a Pipeline needs to be usable by more than one Assistant — a strict tree can't express that without duplicating the resource, and it forces every future concept (a Tool, an Agent workflow) to nest under whichever Assistant happens to touch it first, which breaks the moment two assistants need to share one. The shape that survived instead: Workspace is the one real containment root — already true everywhere else in this codebase, since every table scopes through workspace_id — and KnowledgeBase, PromptCollection, Pipeline, and Assistant are independent, workspace-scoped peers connected by foreign keys, not nesting. Assistant references an active Pipeline, a PromptCollection, and any number of knowledge bases through a join table; it doesn't own any of them. A binding record composes cleanly with whatever gets added later by adding one nullable FK — no redesign required to grow the system.

Underneath that, provider-specific columns (ollama_embedding_model, st_embedding_model, and so on — a pattern that only gets worse with every new vendor) were replaced with a single model_providers table: (kind, provider, model_name) rows for LLM, embedding, and reranker alike. Adding OpenAI, Voyage, or Cohere later is a new row, not a migration. Three more tables — generation_configs, retrieval_configs, index_configs — are append-only: changing a setting never updates a row in place, it inserts a new one and a new pipelines row referencing it, then flips which pipeline is active. Old rows are kept, not deleted, which gives rollback (flip is_active back to a previous pipeline) and A/B comparison for free, with no separate history table. The traceability requirement came out of this almost automatically: every Document row stores the exact IndexConfig that was active when it was ingested — NOT NULL, RESTRICT on delete — re-stamped on completion so a Rebuild Index that spans a mid-flight settings change still records the truth per document, not just what was active when the batch started.

The part that took actual judgment, not just schema design, was reconciling the new configurable knobs with retrieval logic that already had real behavior baked in. Each new setting needed an explicit answer to "does this change what runs, or just how the result gets used" — treated naively, half of them would have quietly changed behavior for pipelines that never asked for it. search_type defaults to "auto", provably identical to the pre-existing behavior for anyone who never touches it. Disabling query rewrite still runs the classification call underneath, since intent is load-bearing for which retrieval strategy dispatches — only the rewritten query text gets discarded. Multi-query retrieval is bounded to 2–5 variants and explicitly excluded from the existing per-entity comparison fan-out, so enabling it can't silently multiply LLM calls. Getting each of these five tensions right mattered more than getting the schema right in the first place — the schema is easy to get right once; the migration path for existing behavior is where a redesign actually goes wrong.

One real bug came out of building the API on top of this: PATCH /admin/rag-settings built its update payload with Pydantic's exclude_none=True, which makes an explicit null in a request indistinguishable from a field the client never touched. That's fine for most fields, but fatal for a nullable one — there was no way to ever clear reranker_provider_id back to NULL through the API once it was set, since a null in the request body just got silently dropped instead of applied. A failing test caught it, not a design review. The fix was switching to model_dump(include=payload.model_fields_set), which reads which fields were actually sent, not their values — "omitted" and "explicitly null" stopped being the same thing.

How the Hard Problems Were Solved

A Role Hierarchy That Could Lock Out Its Own Admins

RBAC here started as a single binary flag: platform admin or not. That doesn't survive a real request — the ability to build an arbitrary role hierarchy from the UI, with a super_admin that's untouchable, an admin with equal power that still can't touch super_admin, and a manager below both, none of it hardcoded. The real incident that forced the redesign: a user deleted the seeded platform_admin role, created a custom super_admin in its place, and was immediately locked out of the entire admin UI — including the Roles page, the only place a replacement role could have been granted. The root cause wasn't the deletion itself; it was that the frontend had never actually adopted the permission system the backend already computed correctly. Every nav-visibility and page-access check was hardcoded to the literal string "platform_admin", not to the resolved permission set the API already returned — so the moment no role held that exact name, every gated page refused access with no way back in through the UI at all. Access had to be restored directly against the live database while the real fix was built.

The actual fix was a rank column on Role — lower number, more senior, 0 the ceiling — with one rule applied uniformly everywhere a role gets created, edited, deleted, granted, or revoked: an actor can only act on a role whose rank is strictly more junior than their own, never a role at or above their own rank, even one they hold themselves. The deliberate, confirmed consequence: rank 0 becomes permanently frozen through the API the moment it's assigned — nobody, including its own holders, can rename it, re-rank it, edit what it grants, or delete it, because no actor can ever hold a rank below 0. That's exactly the rule that would have prevented the lockout in the first place, and it was chosen over a looser "you can still touch your own rank" variant on purpose. It also means an existing admin can never grant that same top rank to a second person through the API either — "a role at your own rank" applies to grants, not just edits — which is why a separate, deliberately unprivileged function (bootstrap_grant_platform_role(), callable only from a direct script or shell, never wired to any HTTP endpoint) exists as the one sanctioned way to create the first admin account. Creating the first admin was never going to be a self-service HTTP flow anyway; this just gave that necessity an honest name instead of leaving it as ad hoc SQL.

Shipping the rank rule surfaced a second hole almost immediately, this time from an independent audit rather than a live incident: rank alone doesn't stop escalation, because rank is deliberately independent of what permissions a role actually carries. A role holding only platform.users.manage at rank 50 could still grant a different, more junior role at rank 60 that happened to carry a stronger permission, like platform.rag_settings.manage — handing out power the granting actor never had themselves, purely because the target role's rank number was technically more junior. The fix layers a second, independent check on top of rank, not instead of it: granting a role now also requires the actor to already hold every permission that role carries. Revoking doesn't need the same check — taking a permission away from someone else can't escalate the actor's own power, so there's nothing to guard against on that side.

A Vector Index That Can Only Hold One Dimension

After switching the platform's active embedding provider from a 1024-dimension Ollama model to a smaller, faster 768-dimension sentence-transformers model, every subsequent document upload started failing:

WeaviateInsertManyAllFailedError: new node has a vector with length 768.
Existing nodes have vectors with length 1024.

Chunking and embedding were both succeeding — the traceback pointed at the final vector-store write. The cause was structural: Weaviate locks a collection's vector index to whatever dimension its first inserted object had, and the collection still held vectors from the old model. Every new document computed valid 768-dimension vectors and got rejected wholesale by an index that could only accept 1024. The existing "Rebuild Index" admin action couldn't fix this either — it deleted and re-ingested one document's chunks at a time, so as long as any other still-unprocessed document's old-dimension vectors remained in the same shared collection, every reinsert failed the same way. The fix had to be structural too: the rebuild job now wipes and recreates the entire collection once, up front, before reprocessing anything, instead of per document. I verified it against the actual broken data, not just a test — triggered the fixed rebuild and watched every failed document go from failed to completed with real chunk counts.

A Security Check That Silently Never Ran

An independent audit flagged that nothing stopped the app from booting with the shipped-in-code default JWT secret still in place — a misconfigured deployment would silently accept forged tokens for any user, including ones that don't exist. The fix looked simple: refuse to start if APP_ENV != "development" while the secret is still the default. Verifying the other half of the same finding — a debug stack trace leaking to callers on an unhandled exception — is what turned into the real problem. Registering a catch-all exception handler didn't work. Requests that should have returned a generic 500 body still came back with Starlette's raw HTML debug page. The cause was in framework internals, not application code: Starlette's ServerErrorMiddleware checks its own debug flag and renders the debug page before it ever consults a registered exception handler — a debug=True app silently bypasses any custom handler entirely, regardless of whether one exists. The only real fix was flipping that flag's default, with the registered handler as a second, now-actually-effective layer rather than the whole fix. Nothing about this would have surfaced from reading the exception-handling code in isolation; it only showed up by actually triggering the failure and checking what the caller received, not what the handler intended to send.

What I Learned

The hardest problems here were never in the RAG logic itself — they were in the shape of the data underneath it, and in what happens when two supposedly-independent safeguards turn out to depend on each other in a way nobody stated out loud: a rank system that was correct on its own terms but didn't account for the permissions a role actually carries, a registered exception handler that was correct on its own terms but sat downstream of a framework flag that silently overrode it first. Getting the pipeline registry's schema right took three rejected designs before the fourth one actually composed with a future that didn't exist yet. Getting the RBAC rule right took a real lockout to prove which "obviously fine" variant wasn't. Both took longer than writing the retrieval logic did, and neither would show up in a demo — which is exactly why they were worth doing properly the first time.