Multimodal RAG: mixed files still become text
Multimodal RAG over business files means a PDF, a slide, a screenshot, a table, and a recording have to become passages a text retriever can find. OCR, transcription, and a preserved table do that work. The embedding is of the resulting text, and each passage keeps the file and the place it came from.
How does multimodal RAG retrieve mixed documents?
Multimodal RAG retrieves mixed documents by converting each kind into text: a text layer is parsed, a scan is OCR'd, a recording is transcribed, and a table stays a table. Those passages are what the retriever searches and what a citation points at.
The files are mixed, and the questions are not
A real corpus is a PDF, a slide deck with screenshots, a spreadsheet, and a meeting recording about the same deal. The question does not arrive labeled by format. "What did we agree about the renewal" might be a sentence in the deck, a number in the sheet, or a minute in the recording. Multimodal RAG is the name for retrieving across that mix. It is not a promise that one model looks at every pixel and every waveform at question time.
The retriever searches text
ragconverter.com embeds the text of each passage and writes that vector into JSONL, with the manifest naming the embedder. A chart that never became words is not a visual vector you can search by appearance. It is a skip, named as a skip, which is better than a placeholder that matches every query. What JSONL is for RAG is the file those passages live in. If you later want a multimodal embedder, the text is in the same object and the original file is still yours.
Each kind of file has its own conversion
A text-layer PDF, a Word file, and a Markdown note are already words. A table has to stay a table, because a row of numbers detached from its headers retrieves as noise. A PowerPoint is headings plus whatever was pasted on the slide: the prose converts, and the screenshot needs OCR. An image or scan is pixels until OCR reads it. Audio is a transcript with timestamps, or it is not searchable as text at all. Video needs the same split - frames and speech - and a browser cannot demux it.
Order and citations have to survive the mix
A slide's title, the screenshot on that slide, and the sentence under it are one argument. Conversion that runs every image after every paragraph sorts the screenshot to the end of the deck and then retrieves it with no heading. Passages need the source file, the page or the timestamp, and the heading they sat under, so an answer can point at the slide or the minute. That metadata is the difference between a corpus and a bag of strings.
Where the tab is enough
Text-layer documents convert in the browser with no upload. Image OCR can run there too, until a large page presses on WebAssembly's memory limit. Speech and video, and OCR at full resolution, are the API: multipart POST to /v1/convert, same chunk objects, manifest listing anything skipped. Start with the format that is actually in the folder - a scan is an OCR problem, a recording is a transcript problem - and read a sample from the middle before you index the library. The engines for those formats are Remade with Rust.