Matching & blocking

Matching asks, for a pair of records: same person? The naive plan — compare every record with every other — dies on arithmetic: 460 records is a manageable 105k pairs, but 1M records is half a trillion pairs. Resolution at scale is therefore two moves: blocking (only compare pairs that share some cheap signal) then scoring (spend real comparison effort inside blocks only).

Step zero: normalize like you mean it

• Emails: lowercase, trim. (Deeper waters exist — gmail dot-insensitivity, plus-addressing — note them, don't drown in them.)
• Phones: strip everything but digits, keep the last 10. All three of Harbor Lane's POS formats collapse to one canonical string. Normalization alone typically recovers a third of your matches.
• Names: lowercase, collapse whitespace. 'ELENA DIAZ' → 'elena diaz'. Accents and nicknames (José/Jose, Bob/Robert) are the next tier — real systems carry lookup tables for them.

Blocking: the feasibility trick

A blocking key is a cheap hash of a record that probable-matches share: normalized email; phone digits; (last name, zip). Records land in a block per key; you compare within blocks only. A record with an email, a phone, and a name lands in three blocks — three chances to meet its duplicate, no chance of meeting the 999,000 strangers. Blocking trades a sliver of recall (duplicates sharing no key are never compared) for a factor-of-a-million cost reduction. Every production resolver — including Meld — is built on this trade.

Scoring inside blocks

def score(a, b):
    s = 0.0
    if a['email'] and a['email'] == b['email']:
        s += 0.6                     # a shared email is nearly decisive
    if a['phone'] and a['phone'] == b['phone']:
        s += 0.3                     # shared phone: strong, families blur it
    s += 0.3 * name_similarity(a['name'], b['name'])   # fuzzy, 0..1
    return s                         # match if s >= threshold (e.g. 0.6)

• Weights encode evidence strength. Email exact ≈ decisive. Phone exact is strong but families share phones. Name similarity is weak alone — the world contains many John Smiths — but decisive in combination.
• Fuzzy name comparison uses string similarity (edit distance, or Python's built-in difflib.SequenceMatcher) so 'Jon Smith' ≈ 'John Smith' scores ~0.9 instead of 0.
• The threshold is a business dial, not a constant of nature. Too low → false merges (two people become one — the worst failure: wrong data served confidently, hard to unwind). Too high → missed matches (the status quo you came to fix). Production systems use two thresholds: auto-merge above the high bar, human review queue in the gray zone.