How to evaluate document parsing for RAG
You evaluate document parsing for RAG by scoring the parse against your own files on several dimensions - text fidelity, tables, reading order and what happens on failure - because a single accuracy figure hides the errors that actually poison retrieval.
A headline score cannot tell you where it will fail
Unstructured, LlamaParse and Reducto all publish parsing numbers, and they are measuring real things. They are also measuring their own corpora. Your 10-Ks, your scanned claims forms and your internal decks are not in those sets. A parser that leads a public table on financial filings can still scramble the two-column research notes that make up most of your library. Evaluate on a sample of the files you will actually convert.
Score four things, not one
Text fidelity: did the words come out as written, including hyphenation and footnotes. Table structure: do headers still sit on their columns. Reading order: does a two-column page come out as two columns, or as interleaved sentences. Silence: when a file cannot be read, is it reported or indexed as an empty passage. The last one is the one vendor tables omit, and it is the one that quietly wrecks a corpus - placeholder chunks are nearly identical and outrank real answers.
Separate parse, retrieve and generate
If the answer is wrong, the instinct is to change the prompt. That only helps when the right passage was retrieved. If retrieval missed, the parse or the chunker is at fault and no prompt will find a sentence that was never formed. Keep a short list of questions whose answers you can point to on the page, and record three bits: was the passage extracted at all, was it retrieved, and did the model use it. That attribution chain is the whole of a useful RAG evaluation.
A manifest is the instrument
You cannot score what you cannot describe. A conversion should say how many files it read, which it skipped and why, how many chunks each file produced, and whether the text was parsed, OCRed or described. RAG Converter writes that to manifest.json next to the JSONL. An empty yield on a file you expected to convert is the fastest accuracy test there is, and it does not require a labelled dataset to run.