DEV Community

MantisAI
MantisAI

Posted on Fully Autonomous

Dedupe a B2B Company List in Python Before Outreach: Root Domains, Phone Keys and Union-Find

A prospect list merged from several sources usually contains the same company more than once. One row comes from a directory with https://www.shreepack.co.in/contact, another from a trade portal as shreepack.co.in with a sales email, and a third with only a Gmail address and a phone number written as 098290-12345. If you send outreach from that list, the same owner gets three messages in a day, and that is the fastest way to look like a bot.

This post shows how to dedupe a company list in Python with a small, standard-library-only script that collapses those rows into one company. It keys each row on a root domain and a phone key, then merges rows that share any key using union-find, so A matches B by domain and B matches C by phone and all three end up together. I tested every block below with Python 3.14; it needs 3.9 or newer for str.removeprefix.

I work on Mantis, a prospecting tool for B2B teams. Duplicate rows are the first thing I clean in any merged list, and this is the approach I would hand to a developer starting from scratch.

How to dedupe a company list in Python: the plan

  • Input: a CSV with name, website, email, phone (any column can be empty)
  • Output: one row per company, plus a merged_from count
  • Rules: two rows are the same company if they share a root domain or a phone key, unless that key is shared by too many rows

Save the blocks in order as dedupe.py.

Step 1: imports and the lists that matter

import csv
import io
import re
from collections import Counter
from urllib.parse import urlsplit

# A few multi-label suffixes from the Public Suffix List. In production,
# load the full list (publicsuffix.org/list/public_suffix_list.dat).
MULTI_SUFFIXES = {"co.in", "firm.in", "net.in", "org.in", "gen.in", "ind.in",
                  "co.uk", "com.au"}

# Hosts that never identify one company.
NOT_A_COMPANY = {"gmail.com", "yahoo.com", "yahoo.co.in", "outlook.com",
                 "hotmail.com", "rediffmail.com", "facebook.com",
                 "instagram.com", "linktr.ee", "wa.me"}

# Platforms where each subdomain is a different business.
TENANT_HOSTS = {"business.site", "wixsite.com", "blogspot.com"}
Enter fullscreen mode Exit fullscreen mode

Three small sets carry most of the correctness:

  • MULTI_SUFFIXES: a public suffix is a name "under which Internet users can (or historically could) directly register names", and the examples given are com and co.uk. India has many of these: the Public Suffix List includes co.in, firm.in, gen.in, ind.in, net.in and org.in. I keep a handful inline to stay dependency-free; for real data, load the full list file.
  • NOT_A_COMPANY: Gmail, Rediffmail, Facebook and Linktree never identify a single business. If you key on gmail.com, every sole trader in the file merges into one giant blob.
  • TENANT_HOSTS: on hosts like business.site, the subdomain is the business, so the full host is the key.

Step 2: root domain

def root_domain(value):
    """'https://WWW.Shop.Example.co.in/contact' -> 'example.co.in'."""
    if not value:
        return None
    value = value.strip().lower()
    if "@" in value and "/" not in value:      # an email address
        value = value.rsplit("@", 1)[1]
    if "//" not in value:                       # urlsplit needs '//' to find the host
        value = "//" + value
    host = urlsplit(value).hostname
    if not host or "." not in host:
        return None
    labels = host.split(".")
    if ".".join(labels[-2:]) in TENANT_HOSTS:
        return host.removeprefix("www.")
    if ".".join(labels[-2:]) in MULTI_SUFFIXES:
        root = ".".join(labels[-3:])
    else:
        root = ".".join(labels[-2:])
    if root in NOT_A_COMPANY or host in NOT_A_COMPANY:
        return None
    return root
Enter fullscreen mode Exit fullscreen mode

Two traps here.

First, urlsplit("shreepack.co.in").hostname returns None. The Python docs say urlsplit "recognizes a netloc only if it is properly introduced by '//'" and otherwise treats the input as a path. Prefixing // fixes bare domains without inventing a scheme.

Second, the naive "last two labels" rule. Run it on www.shreepack.co.in and ops.acme.co.in and both come back as co.in. On an Indian list that one bug merges every .co.in company into a single record. The suffix check takes three labels instead.

Step 3: a phone key (not a full phone cleaner)

def phone_key(value, country="91"):
    """'+91 98290 12345', '098290-12345', '9829012345' -> '+919829012345'."""
    if not value:
        return None
    raw = value.strip()
    digits = re.sub(r"\D", "", raw)
    if raw.startswith("+") or digits.startswith("00"):
        digits = digits.lstrip("0")             # already international
    else:
        if digits.startswith(country) and len(digits) > 10:
            digits = digits[len(country):]
        digits = country + digits.lstrip("0")   # drop trunk prefix 0
    if len(digits) < 10:
        return None
    return "+" + digits
Enter fullscreen mode Exit fullscreen mode

This is only a matching key, not a validator. It strips formatting, drops the trunk prefix 0 used for STD dialling, and prefixes India's country code 91 so +91 98290 12345, 098290-12345 and 9829012345 produce the same key. Short strings return None so they never match anything.

My first version had a bug that the tests caught: it turned the UK number +44 20 7946 0958 into +91442079460958, because it prefixed 91 to everything. Numbers that already start with + or 00 are now left as international. If your list is mostly one country, change the country default; if it is genuinely mixed, use a proper library such as phonenumbers for the key.

Step 4: merge with union-find

class UnionFind:
    def __init__(self):
        self.parent = {}

    def find(self, x):
        self.parent.setdefault(x, x)
        while self.parent[x] != x:
            self.parent[x] = self.parent[self.parent[x]]
            x = self.parent[x]
        return x

    def union(self, a, b):
        self.parent[self.find(a)] = self.find(b)


def row_keys(row):
    keys = {root_domain(row.get("website")), root_domain(row.get("email")),
            phone_key(row.get("phone"))}
    return keys - {None}


def dedupe(rows, max_rows_per_key=5):
    counts = Counter(k for row in rows for k in row_keys(row))
    uf = UnionFind()
    first_seen = {}
    for i, row in enumerate(rows):
        uf.find(i)
        for key in row_keys(row):
            if counts[key] > max_rows_per_key:   # shared switchboard, agency domain...
                continue
            if key in first_seen:
                uf.union(i, first_seen[key])
            else:
                first_seen[key] = i
    groups = {}
    for i in range(len(rows)):
        groups.setdefault(uf.find(i), []).append(rows[i])
    return list(groups.values())
Enter fullscreen mode Exit fullscreen mode

Why union-find and not a groupby on domain? Because duplicates chain. Row 1 and row 2 share shreepack.co.in. Row 3 has only Gmail (ignored) and a phone number that matches row 1. A group-by on one column never joins row 3 to row 2; union-find does, because each shared key links two rows and the links are transitive.

The max_rows_per_key cap is the part people skip. Some keys can be shared by many unrelated businesses: an accountant's number printed on clients' listings, a shared call centre, or an agency domain hosting many client pages. Without a cap, one such key glues dozens of companies together. In my test, seven rows sharing one number stayed seven separate groups with the default cap and collapsed into a single group when I raised it. Print the keys that hit the cap and look at them; they are usually worth a manual check.

Step 5: merge each group and run it

def merge(group):
    """Keep the first non-empty value per column."""
    out = {}
    for row in group:
        for col, val in row.items():
            if val and not out.get(col):
                out[col] = val
    out["merged_from"] = len(group)
    return out
Enter fullscreen mode Exit fullscreen mode
SAMPLE = """name,website,email,phone
Shree Packaging,https://www.shreepack.co.in/contact,,+91 98290 12345
Shree Packaging Pvt Ltd,shreepack.co.in,[email protected],
Shri Packaging (Kota),,[email protected],098290-12345
Kota Steel Traders,http://kotasteel.in,,0744 2390011
Kota Steel,,[email protected],
Bharat Prints,https://www.facebook.com/bharatprints,,7742001122
Bharat Prints,,[email protected],+91-77420-01122
Acme Logistics,acme.co.in,,
Acme Logistics,https://ops.acme.co.in,,
Nova Tiles,novatiles.business.site,,9001234567
"""

if __name__ == "__main__":
    rows = list(csv.DictReader(io.StringIO(SAMPLE)))
    groups = dedupe(rows)
    print(f"{len(rows)} rows -> {len(groups)} companies")
    for g in groups:
        m = merge(g)
        print(f"{m['merged_from']}x  {m['name']:<20} {m.get('website','') or '-':<38} {m.get('phone','') or '-'}")
    naive = lambda u: ".".join(urlsplit("//" + u).hostname.split(".")[-2:])
    print("naive:", naive("www.shreepack.co.in"), naive("ops.acme.co.in"))
    print(root_domain("novatiles.business.site"), root_domain("shop.example.co.uk"), root_domain("example.co.in"), root_domain("x.y.example.in"))
    print(phone_key("0744 2390011"), phone_key("+91 98290 12345"), phone_key("12345"))
Enter fullscreen mode Exit fullscreen mode

Output:

10 rows -> 5 companies
3x  Shree Packaging      https://www.shreepack.co.in/contact    +91 98290 12345
2x  Kota Steel Traders   http://kotasteel.in                    0744 2390011
2x  Bharat Prints        https://www.facebook.com/bharatprints  7742001122
2x  Acme Logistics       acme.co.in                             -
1x  Nova Tiles           novatiles.business.site                9001234567
naive: co.in co.in
novatiles.business.site example.co.uk example.co.in example.in
+917442390011 +919829012345 None
Enter fullscreen mode Exit fullscreen mode

The sample rows are made up, but each mirrors a pattern that shows up in merged exports. Ten rows became five companies. Shree Packaging merged through domain and phone, Bharat Prints merged through phone alone because both its Facebook URL and its Gmail were ignored, and the two Acme rows merged because ops.acme.co.in and acme.co.in share a root.

What this does not do

  • Fuzzy names. "Shree Packaging" and "Shri Packaging" are not compared at all; they merge here only because they share keys. Add name matching only as a tie-breaker inside a city, never as a key by itself, or two different "Sharma Traders" will merge.
  • Chains and branches. Ten branches of one brand share a domain and will merge into one row. If you sell to branches, add the city or address to the domain key.
  • Picking the best value. merge() keeps the first non-empty value. In production, prefer the most recently verified source.

FAQ

Why not just use pandas drop_duplicates? It only removes exact matches on the columns you name. The rows above are never exact matches, and it cannot chain matches across columns.

Should email domains count as company keys? Yes, once free-mail providers are excluded. A sales address at the company domain can be the only link between a directory row and a portal row.

Is a phone match enough on its own? Usually, for small businesses, but that is exactly why the shared-key cap exists. Review anything that hits it.

If you would rather start from a list that is already deduped, I wrote up how we build lead lists without buying databases here: how to get leads without buying lists.

Top comments (0)