RAG deletion tests: can removed documents still answer?
A RAG deletion test removes a known source and verifies that its passages no longer appear in permitted retrieval or cached answers. Test the whole application path, not just the vector database response, and keep access changes distinct from physical deletion.
How do I test that a removed document stops answering in RAG?
Remove a synthetic source with a distinctive fact, then check its chunk identifiers, filtered retrieval and application caches. A delete response alone does not prove the end-to-end result; record when the source stops appearing for each relevant user.
Make the test observable before deleting anything
Create a synthetic document with a harmless distinctive fact that appears nowhere else in the test corpus. Ask a narrow question whose answer requires that fact. Record the source identifier, resulting chunk identifiers, index or namespace, retrieval settings and the application version. Confirm that an authorized test user can retrieve the expected passage before removal. If the baseline never finds it, a later empty result proves nothing about deletion.
Use synthetic data and an isolated test environment. Do not start by deleting a customer contract or disabling a real user. A deletion test is an acceptance procedure for your application, not permission to operate someone else's corpus. The citation guide explains why a passage needs a traceable source. That trace is also what lets you name every piece a removal should affect.
Follow the source, not just one convenient chunk
A converted file can produce several passages. Keep a mapping from the source version to all of its indexed chunk identifiers, plus any saved conversion output and application-side artifacts. Identify the owning system for each copy before defining what “removed” means. A source disappearing from a folder does not, by itself, demonstrate that an index, cached response or downloaded export has changed.
RAG Converter supplies a conversion seam; this article does not claim it deletes records from your vector database or purges your application's caches. The underlying Remade with Rust ecosystem is context for the conversion stack, not an end-to-end deletion guarantee. The incremental indexing article covers updating changed material. This test adds a different acceptance question: after removal, can the old material still be retrieved?
Distinguish an accepted operation from an observed result
Read the semantics of the datastore you actually use. For example, Qdrant's delete-points reference supports selecting points by identifiers or a filter and exposes a wait parameter. That is the contract of a specific backend operation. Do not translate it into a promise that every copy in a larger RAG application has already disappeared.
Record the deletion request's result, then run a direct lookup for the known chunk identifiers and a retrieval query for the distinctive fact. Repeat observations according to that backend's documented behavior, without inventing a universal settling time. If the operation reports success but the application still returns the passage, retain both observations. A green request log is not a substitute for a retrieval result. No database operation was run to prepare this article.
Test access removal as a separate path
Revoking permission and physically deleting a document are different operations. A retained file can be inaccessible to one person while remaining available to another authorized person. Define which outcome each test expects. Exercise the real authenticated application path, not a privileged administrator search that bypasses the relevant filter.
Microsoft's security-trimming example illustrates filtering results using principal identifiers stored with documents. Its sample pattern is not automatic authorization for every RAG stack. Your application must supply and enforce the correct user context. Test both a user who lost access and a user who should retain it; a result that hides the document from everyone may be a broken application rather than a successful permission update.
Inspect caches and answers without confusing recall with proof
Repeat the distinctive question through the same interface a user normally sees. Check returned passages and citations as well as the generated answer. Inspect the caches your own architecture actually has, such as retrieved-result or answer caches; do not invent components merely to fill a diagram. A newly generated response that contains the old fact needs investigation, but text alone cannot identify which layer retained it.
A model might produce a similar answer without retrieving the deleted source. That is why the baseline uses a synthetic distinctive fact and why the evidence includes retrieved identifiers, not only prose. Conversely, removing a citation label while retaining the underlying passage is not a deletion success. Keep logs appropriately redacted and do not turn the test report into another uncontrolled copy of sensitive source content.
Write a result that another developer can reproduce
The report should say which source version was removed, where its chunks were stored, which users and queries were tested, what each layer returned and when each observation happened. Mark failures, unknowns and layers you could not inspect. Retention obligations, backup handling and service-specific deletion policies need their own decisions; a retrieval test is not a legal erasure certificate.
Start by converting a harmless representative document at the converter, then retain its source-to-chunk mapping in your own test harness. The developer API guide is relevant when conversion must run in a pipeline, but this article promises no automatic downstream cleanup. If you also evaluate MATA's wider ownership approach, the Digital Freedom roadmap is the context, not evidence that it administers your existing index. Finish with a reproducible observation, not the sentence “the delete returned OK.”