RCRAG Converter

Vector search microbenchmarks are not RAG retrieval evaluations

A vector database microbenchmark measures the cost of a fixed search, while a RAG retrieval evaluation measures whether the right passages come back for real questions, so a downstream database upgrade needs its own retrieval check before anyone calls retrieval better.

Does a faster vector search benchmark mean better RAG retrieval?

No. A vector microbenchmark counts the work of a fixed search on fixed data. RAG retrieval quality depends on chunking, embeddings and the questions asked. After a downstream database upgrade, re-run a question set with known answers and compare the returned passages, not only the timing.

What a vector microbenchmark measures

A microbenchmark fixes the data and the query and measures the cost. SpaceDB, an open-source local-first database, is a current example. Its 0.7.2 release reports that a vector driver of 1,500 vectors with 64 dimensions, three distance metrics and top-10 search fell from 33.1 million to 23.3 million instructions, 29.8 percent, against a re-measured 0.7.1 (commit). A query driver fell 18.0 percent. Those are repository-reported callgrind counts on synthetic drivers, with no wall-clock time reported.

That is useful engineering information. It tells you the search does less work for that shape of data. It does not tell you whether the passages your users need are more likely to come back. RAG Converter does not use SpaceDB, and nothing in this article describes a built connection between them.

Why a faster search can return different results

The same SpaceDB release changed how vector scores are summed: dot products and distances now accumulate in eight lanes and fold in a fixed order. Floating-point addition is not associative, as Goldberg's classic paper explains, so a score can differ from the previous version in its last bits. The project reports that rankings in its own test corpus did not change and that two near neighbors within that rounding could swap places.

For a RAG corpus that means a database upgrade can move which chunk sits at rank five, even when no document changed. Usually that is harmless. Sometimes it is the difference between citing the right clause and the one next to it. Only a retrieval check on your own data will show which case you are in.

Run a retrieval check after every hand-off

Keep a fixed question set with the passage that should answer each question, built from your own converted documents. When the vector store, its version, the embedding model or the chunker changes, run the set and record which passages come back in the top results, how often the expected passage appears, and whether the order changed. The hybrid search guide covers one reason order can move, and the deletion test uses the same habit of recording returned identifiers rather than trusting a status message.

Store the database name and version, the embedding model and the converter output version with each run. If a microbenchmark improved and your retrieval set is unchanged, you have a cheaper search and the same quality, which is a perfectly good result. If the set moved, investigate before you call the upgrade an improvement.

Keep each claim in its own lane

The converter's job ends at the hand-off: its JSONL chunks go into a store you run. RAG Converter's own persisted corpus and query API is planned, not available, so retrieval quality after the hand-off is measured in your pipeline, not on this site. SpaceDB's vector index returns ids and scores and keeps stored vectors on its node; that is a SpaceDB interface, not a RAG Converter feature.

A database release note is evidence about the database. A retrieval evaluation is evidence about your corpus. Both are worth having, and neither substitutes for the other. The libraries behind the converter come from Remade with Rust, which is also where SpaceDB is built, and that shared ecosystem is context, not a claim that the two are integrated.

More on converting for RAG