RCRAG Converter

Duplicate documents in RAG: find the copies before you index

Duplicate documents in a RAG corpus, such as one file saved in three folders or five drafts of the same policy, fill retrieval with near-identical passages and blur which source a citation should name, so find them before indexing: hash files for exact copies, group near-duplicate versions, and keep one canonical document per group.

How do I remove duplicate documents before RAG indexing?

Hash every file with SHA-256 to find exact copies, then group near-duplicates such as drafts and renamed exports, and keep one canonical version per group. Convert only those files. RAG Converter drops exact repeats inside one document, but it does not de-duplicate across documents, so the cross-file check is yours.

Why duplicate documents crowd out the answer

Retrieval returns the passages that sit closest to a question, usually a handful of them. If the same paragraph lives in three copies of a file, all three copies score the same, and they can take three of those few slots. The model then reads one fact three times and never sees the passage that would have added something new. Near-duplicates do a quieter version of the same thing: a policy saved as v2, v3 and final produces passages that differ by a sentence and compete with each other.

Citations suffer too. When two files hold the same text, the answer can cite either one, and a reader who opens the older draft may find a clause that was removed months ago. The problem is well studied at larger scale. Researchers who examined language model training data found many near-duplicate examples and long repeated substrings in widely used datasets. A shared drive is smaller, but it was assembled the same way: by many people saving copies.

What RAG Converter removes, and what it keeps on purpose

The converter already handles one kind of duplicate. Exact repeats within a single document are dropped, and the count is recorded in `manifest.json`. A running header on a forty-page PDF would otherwise become forty identical chunks that all rank for the same query.

Across documents, nothing is de-duplicated, and that is deliberate. The same paragraph in two files becomes two passages, because a citation has to name the right file and the converter cannot know which copy you consider the real one. Each chunk carries its `source` and `doc_id` in the metadata, as the JSONL guide shows, so a duplicate is visible after conversion. Deciding which copy stays is a choice about your documents, and it belongs before indexing. The citations guide explains why the source name matters once an answer quotes it.

Find exact copies with a file hash

Start with the cheap, certain check. A SHA-256 digest is a fingerprint of a file's bytes, defined in NIST's Secure Hash Standard (FIPS 180-4), which describes digests as a way to detect whether content has changed. Two files with the same digest have the same bytes. On Linux run `sha256sum` over the folder, on macOS `shasum -a 256`, and on Windows `Get-FileHash` in PowerShell, then sort by digest and look for repeats.

A different digest does not prove different content. Re-saving a PDF or exporting a Word file again can change bytes and metadata while the text stays the same. For those, compare the extracted text instead: convert the folder, download `documents.jsonl`, which holds one row per file with the whole document as ordered markdown, and look for rows whose text matches even though the files did not.

Group near-duplicate versions before you choose

Near-duplicates are the harder case: drafts, renamed exports, the same contract with a different signature page. Search engines have treated this as its own problem for a long time, as Google's 2007 paper Detecting near-duplicates for web crawling shows. For a business folder of a few hundred files, a person with a sorted list usually does better than a threshold. Group files whose names, sizes and first headings line up, then open the group and compare.

For each group, keep one canonical version: the approved one, or the latest one with a named owner. Move the rest out of the ingestion folder rather than deleting them, so the history still exists where people expect it. Write the choice down in a short list beside the corpus, with the digest of the file you kept.

Keep the corpus clean as it grows

Duplicates come back, because people keep saving copies. The digest list you made is also the start of an update routine: when a file's digest changes, it is a new version, and the incremental indexing guide covers reconverting only what changed. When a new file arrives with a digest you already have, it is a copy, and it can stay out.

If duplicates were already indexed, removing them is a deletion, not a conversion. The converter does not delete records from your vector database. Remove the superseded copy's chunks there, then check that its passages no longer come back, using the method in the RAG deletion test.

Start with one folder

Pick the folder people save into most often, hash it, and group what you find. Then convert only the canonical files at the converter and read `manifest.json` for the in-document repeats it dropped. Browser conversion is a Preview capability, so check its current status on the site before relying on it for a production pipeline. In the free tier the conversion runs in your browser tab, within 8 MB per file and 400 chunks, and the files never leave the device. The parsing stack comes from Remade with Rust, the memory-safe libraries behind the converter. If the folder is your own archive rather than a team drive, MATA's Home Computer era is about keeping those originals on hardware you own.

More on converting for RAG