DEV Community

Cover image for Hybrid Search, Measured
FDE at Trida AI for Trida AI

Posted on Originally published at trida.ai

Hybrid Search, Measured

Originally published at Trida AI.

Ask a retrieval system about one error code and it may answer from the record for a different one. The two records read almost identically: same service, same shape of incident, codes differing by two transposed digits. An embedding model, which scores meaning rather than characters, can rank the wrong one first. The assistant downstream then writes a confident, fluent answer about the wrong fix. Nobody sees a retrieval error. They see a wrong answer.

This is the standing argument for hybrid search: keep the vector index for meaning, add a keyword index for the strings that must match exactly, merge the two lists. The advice is usually asserted rather than measured, so we measured it, on a corpus we control, with the awkward queries left in.

The test

The corpus was 373 chunks. Most of it is our own writing: 325 chunks cut from fifty of the posts on this blog. The rest is 48 operational records we wrote for the test: incident notes, part records, runbook owners, product notes, policies, host records, seeded with near-neighbours on purpose (error codes differing by transposed digits; a gasket for the P-9 pump and a near-identical one for the P-6; a Dana Whitfield and a Dana Whitmore who own different queues). Real corpora grow these confusions on their own. We planted ours where we could see them.

Thirty-two queries, written and frozen before anything ran. Twelve were exact-term lookups: a bare code, a part number, a name. Ten were paraphrases of passages in our posts, worded to share as little vocabulary with the passage as we could manage. Ten were mixed: an exact term inside a real question, or a choice between two confusables. Each query has exactly one correct chunk, and we scored every method on whether that chunk came back in the top five, and at what rank.

Three retrievers over the same chunks. Vector search with all-MiniLM-L6-v2 embeddings, cosine similarity. BM25, the keyword ranking function, in a small implementation we wrote for the test (k1 = 1.5, b = 0.75). And a hybrid: reciprocal rank fusion with k = 60, merging the two top-50 lists. No reranker and no query rewriting: we wanted the retrieval layer's own behaviour, not a pipeline's.

Query class Queries Vector (top 1 / top 5) BM25 (top 1 / top 5) Hybrid (top 1 / top 5)
Exact-term 12 9 / 12 12 / 12 10 / 12
Paraphrase 10 4 / 6 1 / 4 3 / 6
Mixed 10 7 / 8 9 / 10 8 / 10
All 32 20 / 26 22 / 26 21 / 28

The rig: 373 chunks in one corpus, two retrievers over it, and one fused list from reciprocal rank fusion (k = 60).

The rig: 373 chunks in one corpus, two retrievers over it, and one fused list from reciprocal rank fusion (k = 60).

Exact terms are not a semantic problem

On the exact-term queries, BM25 was perfect: the correct record at rank one, twelve times out of twelve. A code is not a topic: it either matches or it doesn't, and a keyword index knows the difference between 1810 and 1180 in a way a vector does not.

Vector search never lost the record. The correct chunk sat in its top five every time, but a wrong near-neighbour took rank one in three of the twelve. ERR_AUTH_1180, a password-policy lockout, outranked ERR_AUTH_1810, the failover problem the query named. The P-6 pump's gasket outranked the P-9's. A connector error outranked the sync error. Each time, the right record sat at rank two. That sounds like a near miss and behaves like a failure, because the model downstream reads the top chunk first and builds on it — and the top chunk was a different incident with a different fix.

What each method returned for "AX-4410-B gasket", from the run output. The correct record is the P-9 gasket; vector search and the hybrid both put the P-6 gasket first.

What each method returned for "AX-4410-B gasket", from the run output. The correct record is the P-9 gasket; vector search and the hybrid both put the P-6 gasket first.

Plain-word questions humbled everyone

On paraphrases the expected order flipped, then flattened. Vector search found the correct chunk in the top five for six of ten. BM25 managed four. The hybrid matched the vector score, six, and beat neither. If your traffic is mostly people asking questions in their own words, this test gives you no accuracy reason to run a second index.

The worst miss was shared. One question (how a system cautiously tests whether a failed tool has recovered before letting traffic back through) put the correct chunk at rank 126 under vector search and rank 76 under BM25. The hybrid never saw it at all: fusion only merges what the two retrievers surface, and neither surfaced it in fifty places. Asked about the same chunk with the words "circuit breaker" in the question, BM25 and the hybrid both returned it at rank one. The distance between those two outcomes is vocabulary, not meaning, and no blend closes it. Fusion did pull three buried answers closer — vector ranks 13, 25 and 22 became 7, 13 and 15 — still outside the five results a model actually reads. We count that as no rescue.

The blend has a failure mode of its own

The mixed queries are where the hybrid earned its keep: ten of ten correct chunks in the top five, matching BM25 and ahead of vector search's eight. Across all 32 queries it found 28, against 26 for either method alone. That is the honest headline: a modest, real gain, concentrated where the two signals overlap.

But fusion is an average of two opinions, and it can talk a certain answer down. Twice on exact-term queries (the P-9 gasket; ERR_AUTH_1810) BM25 held the right record at rank one and the fused list demoted it to rank two, because the vector list preferred the neighbour. Our read: run the hybrid as the default, but let an exact match on a code-shaped token outrank the fusion when one fires. A part number is not a vote. That rule is a judgment from this data, not a property of the method.

What it costs, and when we would pay it

The cost is not query time. On this rig a vector query took about 9 milliseconds and our from-scratch BM25 scan about 24, numbers that say more about a teaching implementation than about a production inverted index. The real cost is a second index to build, keep in step with the first, and interrogate at 2 a.m. when an answer is wrong and two rankings might be to blame.

Pay it when the corpus carries identifiers: tickets, parts catalogues, policy documents, logs, anything where users paste strings that must match character for character. That describes most production corpora we meet. Postpone it for a pure prose corpus of questions and explanations, where our paraphrase results say the second index buys complexity and returns nothing at the top of the list.

What this test cannot tell you

Thirty-two queries is a direction, not a significance test. One embedding model, and a small one; a larger model may blur codes less, or blur them differently. Our 48 records were written to be confusable on purpose. Real corpora are too, but rarely so neatly. And at 373 chunks every retriever is effectively scanning the whole haystack; at a million chunks the dynamics change. What survives those caveats is the shape of the result: keyword search owns exact strings, vector search owns paraphrase, and a blend buys the overlap while importing a demotion problem you have to design around.

Work with us

Retrieval is usually the first layer we audit in a RAG deployment, because its failures arrive disguised as the model's. If your assistant answers confidently from almost-the-right record, send us the corpus and the questions it keeps missing through Trida AI's intake, and we will show you which layer is actually failing.

Top comments (0)