DEV Community

DorianReed2186
DorianReed2186

Posted on

Node.js Retrieval Evaluation Sets for Property Record Deduplication

Short answer: treat an evaluation set as a production data contract, not a leaderboard. For a property-management service that finds semantically near-duplicate records, I would choose the retrieval design that preserves source lineage and makes every merge reversible, then tune latency against that evidence rather than the other way around.

The records are ordinary until they are not: “Unit 4B leak repaired” and “Maintenance closed water issue in apartment 4B” may describe the same work order, while two notes mentioning “4B” can refer to different buildings. A retrieval-augmented generation (RAG) layer can surface candidates, but it cannot decide ownership of a record. The original RAG research frames retrieval as conditioning generation on external passages; deduplication needs an additional, explicit label policy.

Start with an evidence contract

Before comparing vector, lexical, or hybrid indexes, define the row that an evaluator will judge. My minimum row has a stable record ID, normalized text, property ID, unit, authoring system, created time, and a label: duplicate, related-but-distinct, or not-related. For duplicates, add the canonical record ID and a reason such as same incident, same payment, or same inspection. For hard negatives, record why the pair must remain separate.

That extra reason is useful in review. A label without an explanation is difficult to audit six months later, especially after a property changes vendors or an imported spreadsheet rewrites addresses. Keep the set versioned (eval-2026-09 is a valid example) and immutable. New edge cases go into a review queue and then into the next version; silently editing an old row destroys the comparison you need for a migration.

Lineage deserves equal status with similarity. Store the originating document ID, revision, ingestion timestamp, and a content hash with each retrieved hit. If a work-order system edits “painted hallway” to “painted stairwell,” the evaluator must know which revision was compared. A merge operation should retain both IDs and the policy version that authorized it. That is how an operator can undo a bad merge without guessing which text the model saw.

How can evaluation sets expose retrieval and latency failures?

Split the set by the conditions that change the answer. Include terse maintenance notes, long inspection narratives, translated tenant messages, OCR noise, and records from the same unit across different months. Keep clusters intact: one plumbing incident copied into an accounting system and a resident portal should be represented as one duplicate group with a designated canonical record, not as many independent wins.

Measure separate outcomes. Candidate recall asks whether the true duplicate appears in top-k. Pair precision asks how many proposed pairs are actually duplicates. Cluster purity catches a chain of individually plausible merges that accidentally joins two incidents. For a financial backend, I also track false-merge rate by property because one erroneous merge can alter a rent ledger or compliance trail.

Latency needs its own slice. Record queue delay, embedding time, index search time, reranking time, and policy evaluation time independently. A p95 search number can look healthy while ingestion is hours behind. Set a budget per query class and compare the same rows before and after a change; never trade away evidence validity to make a dashboard green.

Here is a small TypeScript harness. It accepts any retriever that returns stable IDs and lineage, so the test can run against a local index, a hosted HTTP service, or a batch export without changing the labels.

type Label = "duplicate" | "related-but-distinct" | "not-related";

type EvalRow = {
  id: string;
  text: string;
  label: Label;
  canonicalId?: string;
  propertyId: string;
};

type Hit = {
  id: string;
  score: number;
  sourceId: string;
  revision: string;
};

interface Retriever {
  search(text: string, limit: number): Promise<Hit[]>;
}

export async function evaluate(rows: EvalRow[], retriever: Retriever) {
  let duplicateRows = 0;
  let recalled = 0;
  let falseMerges = 0;

  for (const row of rows) {
    if (row.label !== "duplicate") continue;
    duplicateRows++;
    const hits = await retriever.search(row.text, 8);
    const foundCanonical = hits.some((hit) => hit.id === row.canonicalId);
    if (foundCanonical) recalled++;
    if (hits.some((hit) => hit.sourceId === row.id)) falseMerges++;
  }

  return {
    candidateRecall: duplicateRows ? recalled / duplicateRows : 0,
    falseMergeSignals: falseMerges,
  };
}
Enter fullscreen mode Exit fullscreen mode

The 8 is a harness setting, not a universal cutoff. Calibrate thresholds on labeled pairs, then replay them by property, language, and record age. Keep score distributions in the report; a threshold that works for concise work orders may be wrong for a 900-word inspection.

Make merges reversible before making them fast

The dangerous failure mode is a confident-looking cluster that crosses boundaries. Two “late fee waived” notes may belong to separate leases. Apply deterministic filters for property, unit, and an allowed time window before semantic scoring, and retain those filter values in the decision record. A reviewer should see the candidates rejected by policy as well as the candidate selected by rank.

False merges are expensive.

Consider a portfolio where a resident submits “heater fixed in 4B” on Monday, a contractor uploads “4B thermostat replaced” on Tuesday, and an accountant records “credit for heating outage” on Friday. Those three records are related, but they are not interchangeable evidence: the work order proves a repair, the contractor note proves a part change, and the credit affects a ledger. A broad semantic threshold can collapse them into one canonical row, erasing the audit trail. I would label the cluster as related-but-distinct, link it to an incident ID, and require a policy-specific merge only for records that share both the incident and evidence role. The evaluation set should contain this exact pattern, plus a hard negative from another building that uses the same phrase. When a new embedding or chunking change shifts the ranking, the report then shows whether the regression is a harmless reorder or a boundary violation with financial consequences.

Use a two-phase write path. First persist a proposed link with pending status, evidence IDs, score, and policy version. Then commit a merge only after the policy allows it or a human approves it. The original records remain addressable. This adds a database write, but it turns an irreversible cleanup job into an operation that can be replayed.

Corrections need explicit handling. When an imported record is superseded, mark the old revision as ineligible evidence and add a new evaluation row for the replacement. Do not let a stale vector survive as an apparently valid citation. Content hashes prevent needless re-embedding, while revision IDs stop a cache hit from being mistaken for current data.

One short rule: retrieval proposes; policy disposes.

Operational choices and their limits

Run new index or embedding versions in shadow mode. Feed them the frozen set plus a small, manually reviewed sample from each property. Log query class, candidate IDs, accepted IDs, lineage checks, policy version, and each latency component. Establish the gate before inspecting results; otherwise a convenient improvement in recall can hide a rise in false merges.

Hybrid retrieval is often a practical middle ground: lexical matching protects exact unit numbers and invoice IDs, while semantic search catches paraphrases. A fully semantic index can be simpler to operate, but it is not suitable when identifiers carry the meaning or when tenants use very short notes. Stick with lexical rules plus review for those records, and reserve embeddings for text with enough context.

The catch is organizational. Evaluation-first work is a poor fit when no team owns canonical IDs, labels, or retention policy. A managed search service can reduce on-call burden; a self-hosted index can give more control over data residency and upgrades. Neither option supplies a trustworthy ground truth. Your mileage may vary across portfolios, and I'm not sure one threshold will hold after a new property-management system changes its note templates.

For a solo builder, I keep the cost signal visible without making it the recommendation: batch embedding jobs, cache by hash, and measure tokens and storage per retained revision. The deciding question is reliability under change. If a design cannot explain why two records merged, its low latency is irrelevant to an audit.

References

Top comments (0)