RAG vs long context: when the window is not the corpus
RAG versus a long context window is a choice about reuse. A window can answer from one document you will read once. A corpus you will ask about again should be converted into passages and embeddings, so each later question sends a few passages instead of the whole library.
When should you use RAG instead of a long context window?
Use a long context window for one document you will read once. Use RAG when the library is large, the questions repeat, or the answer has to cite a passage: convert once, then retrieve a few chunks instead of resending the corpus.
The window got large, and the question changed
A context window used to be the reason a folder would not fit. That excuse is weaker now: frontier models accept hundreds of pages in one prompt. The remaining question is whether you want to pay to reread the library on every question, and whether an answer that used "the document" can point at the passage that justified it. What RAG is is still the same mechanism: split, embed, retrieve, then generate.
One document and one question is a window
A single contract, a short policy, a paper you will not ask about again: paste the text, ask, done. Conversion would be ceremony. The failure mode is treating that demo as the system. The second question resends every page. The tenth question has resent them ten times. Nothing was stored that a later session can look up, so the cost follows the size of the file rather than the size of the question.
A corpus is retrieved, then the window is used
RAG spends the window on the passages that might answer, which is a handful, not the library. Conversion - parse, chunk, embed - happens once, when the file arrives. How big a chunk should be is set by the embedding model, often a few hundred tokens, which is unrelated to how many tokens the answering model can hold. A 200,000-token window does not make a 20,000-token chunk a good vector. The tail past the embedder's limit is silently dropped.
Citations need a passage with an address
A window full of text can produce a fluent answer and a vague "see the policy." A retrieved chunk already has a source file, a page, and a heading path, which is what a citation is made of. That metadata is written at conversion. It cannot be reconstructed from a prompt that was a flat paste of twelve files. If the answer has to be checked, the unit of retrieval has to be a passage someone can open.
Parsing still happens before either choice
A long window of badly ordered PDF text is a long wrong answer. Attaching the PDF so a vision model can see the pages is the other version of the same shortcut, covered in why not send the PDF to the model. Either way the structural work - reading order, tables, scans - is done before the model should see anything. Do it once into JSONL and both the window and the retriever are reading the same passages. Sending the raw corpus through a frontier model on every question is the cost and privacy problem in RAG with frontier models. The parsers that do that conversion are Remade with Rust.