RAG in production: the parts that don't show up in the demo

Most RAG demos are the same shape: embed some documents, embed a question, cosine similarity, done. That part works well enough to be convincing in a demo — and it's a small fraction of what a RAG system actually needs to hold up under real use, with real documents, real questions, and real consequences when it's wrong.

Choosing where the vectors live

The honest answer to "which vector database" is rarely about which one is fastest in a benchmark — it's five separate tradeoffs, evaluated together: how much RAM an HNSW index needs at your actual scale, whether hybrid search (keyword + vector) needs to be native or can be approximated, whether there's a relational advantage worth keeping (joining a vector search against the rest of your data, or using row-level security for multi-tenant isolation instead of solving it in application code), how your data actually arrives (streaming inserts favor HNSW, batch loads favor IVF), and the operational cost of introducing a new system versus extending one you already run. Postgres with pgvector is enough for most systems up to tens of millions of vectors — the reason to move past it is a specific, named requirement (native hybrid search, or genuinely massive scale), not a vague sense that a dedicated vector database must be more "serious."

Chunking is a data-modeling decision

A badly cut chunk isn't a cosmetic problem — it's what the model treats as ground truth. Cut a sentence in half, or separate a number from the table row that gives it meaning, and the model has no way to recover that context; it isn't in the text it received. In a document with real structure — a filing with financial tables, headers, and row labels — naively flattening HTML to plain text turns a table into a soup of numbers, and cutting that soup by token count can land a cut mid-row: the figure ends up in one chunk, the period it belongs to in the next. The fix scales with effort: tag chunks that contain a table so they're never split by size; render tables as markdown to preserve row/column association; or extract them as structured data outright when a question needs an exact number, not an approximate match. None of that is optional if the source documents have tables — and metadata (which company, which date, which section) has to travel with every chunk, or nothing stops a chunk from one document bleeding into an answer about a different one that happens to read similarly.

Retrieval beyond similarity search

Cosine similarity finds semantic resemblance, not exact matches — a ticker, a proper noun, or a specific term can get lost if the embedding space doesn't represent it distinctly. Hybrid search runs a lexical method (BM25) alongside vector search and fuses the two rankings — usually by combining rank position rather than raw scores, since the two live on incomparable scales. Reranking adds a second, more expensive pass: a cross-encoder that scores a query against each candidate document together, rather than independently, catching relevance that a fast approximate search alone misses — particularly across content from different sources, where similarity scores from one source often aren't comparable to another's. And query transformation — generating a hypothetical answer and searching by its embedding, instead of the raw question — helps when the way someone asks and the way the answer is written are further apart, in vocabulary and style, than a raw embedding search can bridge. None of these three replace similarity search; a mature retrieval system layers them.

Measuring instead of guessing

"I tried a few questions by hand" doesn't scale and isn't reproducible — change the chunk size, the embedding model, or add reranking, and there's no real answer to whether it helped, only an impression, usually formed by re-testing the easy cases that were already known to work. Production-grade RAG evaluation separates two questions that fail independently: did retrieval bring back the right material (context precision, context recall), and did generation stay faithful to it (faithfulness, answer relevance) — a system can nail one and fail the other. All of this runs on an LLM-as-judge, because there's no formula for "faithful" the way there is for exact-match accuracy, and the judge itself has its own error rate — worth remembering before trusting a small difference between two variants as real. The evaluation dataset is never finished: it starts as a curated set of a few dozen real questions with known answers, and grows every time production surfaces a real failure — including, easy to skip, cases where the correct answer is "I don't have that information," since without testing against those there's no way to know whether the system quietly fabricates when it should say so.

What production adds that tutorials skip

Re-embedding is a cost that's easy to underestimate: reprocessing a document — a chunking change, a bug fix, a new embedding model — means re-embedding every chunk, not just the new ones, since vectors from different models aren't comparable to each other. Reranking is the latency line item that most often surprises people, because a cross-encoder runs once per candidate document per query, and doesn't parallelize as easily as the initial vector search. And changing the index — a new embedding model, different chunking — isn't an incremental insert, it's reprocessing history, which means two "generations" of the same document can coexist mid-migration unless the new version is built in parallel and swapped in atomically, validated against known queries before production traffic ever touches it.

Traceability closes the loop: a citation marker in the output doesn't guarantee the claim is actually backed by that source — verifying it requires the model to cite something checkable (an exact phrase from the source), and a separate, programmatic check that the phrase and the numbers actually exist in that chunk. Skipping that second step is worse than no citation at all — it produces the appearance of verifiability without the substance.

None of this shows up in a demo built on a handful of clean documents and a few friendly questions. It shows up the first time a real document has a table, a real user asks something the eval set didn't anticipate, or a chunking change needs to roll out without corrupting what's already serving traffic.