Your RAG Answers From 62% of Your Corpus and Won't Tell You
Every retrieval system I have ever deployed has one property in common with every retrieval system you have ever deployed: it will answer. It will always answer. Ask it something that lives in a document it never ingested, and it will not say "I do not have that." It will find the nearest thing it does have, rank it, cite it, and hand it to you with the same confidence it would give a perfect match.
This is not a hallucination problem. The model behaved correctly. It answered from what it was given. The failure happened one layer down, in the boring part nobody demos: ingestion.
Last week I measured the corpus coverage of a self-hosted knowledge base I have been running for months. Nineteen hundred documents, semantic search, embeddings served locally, the whole pipeline built and monitored by me. I expected a number close to 100%.
I got 62%.
The number before the number was worse, and also wrong
My first measurement said 8.8%.
That number was real in the sense that a computer produced it. It was also useless, because I had divided the indexed documents by every markdown file on the volume — 13,257 of them — and 11,368 of those belong to entirely different collections that were never meant to be in this index at all.
Same system. Same afternoon. Two measurements. A factor of seven apart.
The honest denominator was 1,890 indexable documents, of which 1,169 were present: 62%. Both numbers were arithmetically correct. Only one described anything.
I want to sit on this for a moment, because it is the part that generalises beyond my homelab. A coverage percentage with an unstated denominator is not a metric. It is a vibe with a decimal point. Every ingestion dashboard I have seen in a commercial product reports documents processed. That is a numerator. It is presented as an achievement — 1,169 documents indexed, green check, ingestion complete — and the question it is quietly not answering is: out of how many, and who decided what counted?
If your vendor cannot state the denominator, your vendor is not measuring coverage. They are counting successes and calling it completeness.
The cause: a notification that never arrives is indistinguishable from nothing to do
The technical root was almost disappointing.
The indexer watched the filesystem with inotify. inotify does not cross NFS. Documents written from the machine running the indexer were seen instantly. Documents written from anywhere else — a laptop, another VM, a sync client — were never seen at all. Not delayed. Not queued. Never.
There is no error state for this. The watcher is healthy. It is watching. It reports zero events, because zero events reached it, and zero events is exactly what a quiet system looks like. Every dashboard stayed green while a quarter of the corpus quietly aged out of reality.
This is the shape of the whole class of bug, and it is worth naming precisely: file events are a notification, not a guarantee, and nothing in the stack is obliged to tell you that the notification never came. Any system built on "react when something changes" inherits a silent failure mode the moment the change happens somewhere the events do not reach. Object storage. Network shares. Container bind mounts. Sync clients that write to a shadow directory and rename. Each one breaks the assumption in its own way, and none of them raise their hand.
What actually fixed it
Two things, neither of them clever.
Polling instead of events. A 60-second sweep calling stat() on the tree. stat() crosses NFS, because it asks rather than waits. It is less elegant and less efficient than inotify, and it works. Verified end to end: a file touched from a different machine now appears in the index ninety seconds later. Before the change, the correct number was never.
Hourly reconciliation. Walk the source tree, walk the index, compare. Emit three counts: documents missing from the index, documents in the index whose source has changed, documents in the index whose source no longer exists. Missing, stale, orphaned.
The division of labour matters more than either mechanism: polling catches up, reconciliation proves. Polling is the repair. Reconciliation is the evidence that the repair worked — and, critically, it keeps producing that evidence after everyone stops paying attention. A fix without a standing measurement is a fix that will silently regress and take another few months to notice.
Coverage went to 1,890 of 1,890. The directory that had been at zero — 235 files of accumulated operational notes, the exact material the system existed to retrieve — was finally in there.
The bug that only showed up once I was looking
Fixing coverage surfaced something considerably worse, and this is the part I would most want a stranger to take away.
The embedding step returned a list of vectors. On failure, that list was compacted — the failed entry dropped. The caller zipped chunks to embeddings by position.
You can see it. One embedding timing out did not lose one section. It shifted every section after it by one. The text of section twelve was stored under the coordinates of section thirteen, and so on to the end of the document.
Searches kept working. Results came back ranked, cited, plausible, and attached to the wrong source. There is no test called "does search return results" that catches this, because search returned results the entire time. The only thing that catches it is deliberately asking a question whose correct answer you already know, and checking the citation rather than the prose.
Related, and cheaper than it sounds: I also excluded roughly 1,500 near-identical boilerplate chunks — template sections repeated across auto-generated reports. They were not missing data, they were crowding. A real query for a specific configuration rule now returns that rule at 0.792 similarity; before, three auto-generated reports sat above it at 0.66. Coverage is not only what is absent. It is also what is present so many times that it drowns the thing you needed.
What to measure on Monday
If you operate retrieval in production, these are cheap and none of them require a vendor:
- Coverage with a stated denominator. Write the denominator down in words. "Indexable" is a policy decision, not a fact — make someone own it.
- Coverage per directory, never as a single average. An aggregate hides a folder at zero. That is not a hypothetical; it is the specific way this hid.
- Staleness and orphans, not just presence. A document indexed at the wrong version is worse than a missing one, because it answers.
- Alignment between text and vector. Store the chunk with its embedding as one unit, or verify the pairing. Never zip two lists by position across a boundary where one can fail independently.
- A canary question. One query per corpus whose correct source you know by heart. Run it on a schedule. Check the citation, not the answer.
The uncomfortable part
None of this was a model problem. Nothing here would have been improved by a larger context window, a better reranker, or a more capable model. The failure was entirely in the plumbing, and the plumbing had no opinion about it.
We spend enormous energy evaluating whether models hallucinate, and almost none verifying that the corpus we are grounding them in is the corpus we think it is. Grounding is only as good as the ground.
A retrieval system cannot report the document it never read. It has no representation of the thing that is absent. Ask it whether its knowledge is complete and it will answer, fluently, from the fraction it holds — which is exactly the question you were trying to settle.
So the question is not whether your RAG works. It answers, so it works. The question is what your coverage number is, what its denominator is, and whether anybody has looked at it since the day it was built.
What is your coverage number — and did you measure it, or inherit it from a dashboard?