RAG for HTML and saved web pages: everything is boilerplate
Converting web pages for RAG is mostly a subtraction problem: navigation, cookie banners and footers repeat on every page and will dominate a corpus unless they are stripped before chunking.
What is the problem converting HTML for RAG?
Converting web pages for RAG is mostly a subtraction problem: navigation, cookie banners and footers repeat on every page and will dominate a corpus unless they are stripped before chunking.
The signal-to-boilerplate ratio is brutal
A typical page is a few hundred words of content inside a few thousand words of navigation, menus, cookie notices and footer links. Convert a site naively and the most common text in your corpus is the main menu, repeated once per page.
Repeated text poisons retrieval specifically
Identical boilerplate produces near-identical vectors. Those chunks then sit close to everything in the embedding space and surface for unrelated queries, pushing the passages that would have answered further down. The damage is not wasted space, it is worse results.
Extract the main content, then chunk
Readability-style extraction - the same idea as a browser's reader mode - identifies the content block and discards the chrome. Doing this before chunking is far more effective than trying to filter the chunks afterwards.
Keep the URL in the metadata
A retrieved passage from a web corpus is much more useful when the answer can be traced back to a page. Carry the source URL and the page title into each chunk's metadata so citations are possible at all. Those format parsers ship as Remade with Rust crates — Office, media and chunking without a C toolchain.