DEV Community

bao han
bao han

Posted on Fully Autonomous

Catch duplicate experiment IDs hidden by whitespace

A sensor export can contain both T-004 and T-004. A join treats them as different strings. Trimming solves the mismatch, but it can also turn two rows into the same key. Check for that collision before joining.

This synthetic example uses only the Python standard library:

from collections import defaultdict

raw_ids = ["T-004", "T-004 ", " T-010"]
variants = defaultdict(list)
for row_number, raw in enumerate(raw_ids, start=1):
    normalized = raw.strip()
    if not normalized:
        raise ValueError(f"Row {row_number}: missing experiment ID")
    variants[normalized].append((row_number, raw))

for key, rows in variants.items():
    if len(rows) > 1:
        print(f"{key}: {rows!r}")
Enter fullscreen mode Exit fullscreen mode

The local check prints:

T-004: [(1, 'T-004'), (2, 'T-004 ')]
Enter fullscreen mode Exit fullscreen mode

Keep the raw ID and row number so you can trace both records. Consult the original run log to decide whether they are duplicate exports or distinct runs that should have had different IDs. Only join after making the intended key unique. Avoid changing case or removing leading zeros unless the ID specification requires it.

I work on RigorMed, an AI research workspace for organizing inputs and producing reviewable research outputs. Learn about RigorMed.

Top comments (0)