Building Search Over a Company Wiki with Retrieval-Augmented Generation

"Just point an LLM at our wiki" undersells how much of the quality in a retrieval-augmented search feature comes from decisions made before the model is ever called.

Chunking strategy decides what the model can actually see

Documents have to be split into chunks before they're embedded and searched, and how they're split matters more than which embedding model is used. Splitting purely by character count tends to cut sentences and sections in awkward places, losing context right at the boundary. Splitting by logical structure — headings, sections, list boundaries — keeps each chunk coherent on its own, which matters because the model only ever sees the chunk, not the surrounding document.

Hybrid search beats pure vector search more often than expected

Vector similarity search is good at finding conceptually related content, but it's often worse than plain keyword search at finding an exact term, a product name, or a specific error code someone is searching for verbatim. Combining both — keyword search for precision, vector search for conceptual matches — and merging the results tends to outperform either one alone, especially on the kind of specific factual questions a company wiki gets asked most often.

The database choice matters less than the pipeline around it

Whether the underlying store is a dedicated vector database or a relational database with a vector extension, the practical differences show up mostly at very large scale or under specific operational requirements. For most internal wiki-search use cases, the retrieval pipeline quality — chunking, hybrid search, re-ranking — has a bigger effect on answer quality than which specific database is doing the vector math underneath it.

Keep the source visible

A wiki search tool that returns a confident-sounding answer with no link back to the source document is harder to trust and harder to verify. Surfacing which document(s) an answer was grounded in — and being honest when nothing relevant was actually retrieved — does more for user trust than any amount of prompt tuning aimed at making answers sound more authoritative.