Core ML

Entity Resolution

Is 'Apple Inc.', 'Apple Computer', and 'Apple' the same company? Entity resolution is the process of looking at messy, real-world data and figuring out which records actually point to the exact same real-world object.

Comparing every record to every other record is O(N²). Blocking groups similar records first, reducing the problem size so we can apply expensive pairwise ML scoring.

Why Does This Exist?

If you merge two massive CRM databases (e.g., Salesforce and Hubspot), you will inevitably end up with duplicates.

Record A: John Doe, 123 Main St, 555-0100
Record B: J. Doe, 123 Main Street, (555) 0100
Record C: Johnathan Doe, null, 555-0100

A simple SQL GROUP BY won't catch these duplicates because the strings aren't exactly identical. But if you don't merge them, your analytics will be wrong, you'll send the customer three copies of the same marketing email, and your recommendation engine will fail.

Entity Resolution (ER) (also called Record Linkage or Deduplication) is the pipeline used to solve this. It is one of the oldest, hardest, and most financially valuable problems in applied data science.

Think of It Like This

Imagine you have a bucket of 10,000 mixed Lego blocks, and you want to find all the identical pairs. If you pick up a block and compare it to every single other block in the bucket, you'll make 50 million comparisons ( $N^2$ ). You'll be there all year. Instead, you first sort the blocks into smaller buckets by Color (Blocking). Now, you only compare red blocks to other red blocks. You might occasionally miss a match (e.g., a faded red block ended up in the orange bucket), but you reduced your workload by 99%, making the problem actually solvable.

How It Actually Works

A modern Entity Resolution pipeline consists of three distinct stages:

1. Blocking (The Filter)

Comparing every row in a 1-million-row database to every other row requires 500 billion comparisons ( $O(N^2)$ ). That is computationally impossible to do with expensive ML models. Instead, we use Blocking. We create simple, fast rules (like "Rows must share the same ZIP code or the same first 3 letters of the Last Name") to group records into "blocks." We only compare records within the same block. This reduces the search space by 99.9%, at the risk of dropping a few true matches.

2. Pairwise Scoring (The ML Model)

Once the blocks are created, we take every pair of records inside a block and score them. We extract features comparing the two rows:

Levenshtein distance between First Names.
Jaccard similarity between Addresses.
Exact match boolean for Phone Number.

We feed these features into a binary classifier (like XGBoost or a Neural Network) trained to output the probability that Record A and Record B are the same entity.

3. Clustering (The Final Decision)

The ML model might say A matches B, and B matches C. But what if it says A does not match C? We have conflicting pairwise scores. To resolve this, we represent the records as nodes in a graph, with the pairwise scores as edge weights. We use clustering algorithms (like Connected Components or Correlation Clustering) to group the nodes into definitive, distinct real-world entities.

Show Me the Code

You can use the open-source recordlinkage library in Python to build a standard ER pipeline.

1import pandas as pd2import recordlinkage3
4# df_a and df_b are two messy DataFrames containing customer data5
6# 1. Initialize the Indexer (Blocking)7indexer = recordlinkage.Index()8# Block on State to avoid comparing a NY resident to a CA resident9indexer.block('state')10# This returns the candidate pairs to compare (drastically reducing N^2)11candidate_pairs = indexer.index(df_a, df_b)12
13# 2. Extract Features for Pairwise Scoring14compare_cl = recordlinkage.Compare()15compare_cl.string('first_name', 'first_name', method='jarowinkler', threshold=0.85)16compare_cl.string('last_name', 'last_name', method='jarowinkler', threshold=0.85)17compare_cl.exact('date_of_birth', 'date_of_birth')18
19# Generate the feature vectors for the candidate pairs20features = compare_cl.compute(candidate_pairs, df_a, df_b)21
22# 3. Pairwise Classification 23# (You could use a trained XGBoost model here, but we use a simple rule-based classifier)24# E.g., if 2 out of the 3 features match, consider it a duplicate25matches = features[features.sum(axis=1) >= 2]26
27print(f"Found {len(matches)} duplicate entities between the datasets.")

Watch Out For

Transitive Closure Errors. In the final clustering step, if A matches B (because they share a phone number) and B matches C (because they share an address), simple connected-components clustering will link A and C together. But what if A is a father and C is his son? They share B (the home phone/address), but they are distinct people. This causes massive "hairball" clusters where thousands of unrelated people get merged into a single mega-entity. Advanced ER requires correlation clustering to cleanly cut these false bridges.

The Quick Version

Entity Resolution merges messy data by identifying which records represent the same real-world object.
Because comparing every record to every other record is $O(N^2)$ , the first step is Blocking to group likely candidates using fast, simple rules.
The second step is Pairwise Scoring, where an ML model looks at fuzzy string similarities to score pairs within a block.
The final step is Clustering, which resolves conflicting pairwise scores into final canonical entities.

Related concepts

Knowledge Graphs