Most RAG projects start the same way: someone embeds a folder of documents, wires a vector store to a model, and gets a demo that answers questions impressively within a day. The demo is real. The conclusion drawn from it — "retrieval works, we just need to add the rest of our documents" — usually is not.
Retrieval-augmented generation fails in production for reasons that have very little to do with the vector database. It fails at ingestion, at evaluation, and at operations. Treating RAG as a system means designing those parts with the same seriousness as the retrieval step itself.
The parts the demo skips
A production RAG capability contains at least six subsystems, and the vector store is the least differentiated of them:
-
Ingestion and normalisation. Documents arrive as PDFs with broken layouts, wikis with stale pages, tickets with half-sentences, and databases with structure the embedder cannot see. Deciding what to extract, how to chunk it, what metadata to preserve, and when to re-ingest is where answer quality is mostly determined — before a single vector exists.
-
The retrieval pipeline. Real systems rarely survive on cosine similarity alone. Hybrid search, metadata filtering, recency weighting, and re-ranking each earn their place depending on the corpus. The pipeline also has to answer a harder question: what happens when nothing relevant exists? A system that cannot say "I don't know" will confidently synthesise an answer from the three least-irrelevant chunks it found.
-
Context assembly. What actually reaches the model — how chunks are ordered, deduplicated, attributed, and trimmed to budget — is a design surface of its own. Attribution matters twice: it lets users verify answers, and it lets you debug retrieval when they can't.
-
Evaluation. Without a test set of real questions scored on retrieval quality and answer faithfulness separately, every change to chunking, embeddings, or prompts is a guess. Evaluation is what turns "it feels better" into an engineering discipline, and it is cheapest to build first, while expectations are still being negotiated.
-
Access control. The corpus almost always contains documents that not every user may see. Permissions have to be enforced at retrieval time, in the index, per query — not by hoping the model withholds what it was given. This single requirement shapes index design more than any performance consideration.
-
Operations. Documents change, embeddings drift out of date, the provider changes model behaviour under you, and costs scale with usage patterns nobody predicted. A RAG system needs freshness monitoring, quality dashboards, and a re-indexing strategy the same way any data platform does.
Why the framing matters
Calling this "a vector database plus a model" leads teams to staff and budget it like a feature. Calling it a system leads them to staff it like a data product with a lifecycle — which is what it is.
The practical consequence is sequencing. The teams that succeed tend to build in this order: a narrow, high-value corpus first; evaluation harness second; retrieval pipeline third; and only then scale the corpus. The teams that struggle do it in reverse — maximum documents on day one, evaluation never.
A note on when not to build it
RAG earns its complexity when the knowledge is large, changing, and genuinely consulted. If the corpus is small and stable, putting it directly in context beats building retrieval infrastructure. If the answers must be exact — prices, entitlements, legal clauses — a deterministic lookup with the model handling only the conversation is both cheaper and safer. The most useful architecture decision in many "we need RAG" conversations is scoping down to the part that actually needs it.
RAG done well is unglamorous: pipelines, contracts, test sets, dashboards. That is also why it works.