My Splink benchmark was wrong four times, every time in my favor

I maintain GoldenMatch, an open-source entity resolution engine. For most of this year I've been benchmarking it against Splink, which is the tool most people reach for when they need probabilistic record linkage, and which is very good.

The headline is that GoldenMatch matches or beats a hand-tuned Splink config on every dataset Splink scores, with no tuning of its own. That's the least interesting thing here.

The interesting part is that I ran the scale comparison five times, four of those runs were invalid, and all four failed in the direction that made my own tool look better.

If you publish benchmarks against someone else's software, that's the failure mode to worry about. Nobody audits the run that agrees with you.

A correction to an earlier post. In April I published a four-library comparison that described GoldenMatch as needing explicit config because "auto-config failed on all three datasets." That was true of the code in April and is not true now: on historical_50k auto-config derives a working config in 2.8 seconds, and every GoldenMatch number below comes from the zero-config path. The April post stands as written rather than being quietly edited, which is the same reason this post exists.

The four bad runs

Splink failed at 50M rows four times before I got a valid comparison. Every failure was my misconfiguration, and every one, published as-is, would have read as a Splink limitation.

One. I set break_lineage_method="persist". Splink's documented default is parquet. Persist caches without truncating the query plan, so the DAG kept growing. Fixing that single line took the 1M-row run from 337s to 104s.

Two. parquet lineage breaking writes from the executors and reads back, so it needs shared storage. My cluster had none. That's not a Splink property, that's me not reading the deployment guide.

Three. Splink started before the Spark master released Connect's cores, and its own wait returned on the first executor to register. It ran on two executors while GoldenMatch had four, and derived 160 shuffle partitions instead of 320.

Four. This is the bad one. spark.executor.memory was unset for Splink, so it took Spark's 1g default while GoldenMatch's Connect server had been launched with 48g. A 48x heap disadvantage.

Defect four caused the 50M OOMs. It also produced a result I'd already half-believed: Splink spilling 4.09 GB of memory and 1.88 GB of disk where GoldenMatch spilled nothing. With equal memory neither arm spills at that size, and a garbage collection gap that looked like 5.9x is 1.19x.

Splink's documented deployments run 100M+ rows on Databricks, EMR and Dataproc, where shared storage and sane executor sizing come for free. Standing a benchmark up on bare Spark and reporting the difference as an engine limitation would have been straightforwardly dishonest, and I was four runs from doing it.

The accuracy side had its own version of this

Before the scale work I published accuracy numbers that turned out not to reproduce. On one commit, three invocations of byte-identical code, same dataset, same runner, same CI run, gave pairwise F1 of 0.805, 0.779 and 0.643. A 0.16 spread with nothing changing between them.

The cause was that the EM training-pair sampler shuffled bare block indices with a seed, but the blocks arrived in a non-deterministic order from parallel construction. The seeded shuffle faithfully permuted a different input every time, so the m and u weights differed, so the threshold differed, so precision and recall differed.

After sorting blocks by a stable key before the shuffle, three separate harnesses agree within 0.002. One previously published figure, 0.879 on a bibliographic dataset, was a lucky draw that doesn't reproduce. The real number on that path is 0.377, which is bad, and it's in the doc.

What the numbers actually say

Every engine is scored by one shared evaluator over the same key space, so all arms are judged by identical code. Splink's configs are genuinely hand-tuned per dataset and reused as written. GoldenMatch runs with no tuning at all.

That table is the August bake-off run recorded in the repo. The timing section below is a separate September run on my laptop, which scores historical_50k at 0.831 rather than 0.827. Same code path, different machine. I'm flagging the seam rather than presenting two runs as one.

Two things belong immediately next to that table.

The 0.97 you've seen quoted for Splink is a cluster-level metric, not exhaustive pairwise F1. Under this harness Splink scores 0.757 pairwise on historical_50k, and that dataset is recall-bound for everyone: 5,156 clusters, mean size around 10, and no single field exceeds 0.60 recall, which puts a ceiling near 0.93 on any engine. The honest claim is "matches or beats Splink under the same evaluator", not "beats 0.97".

Splink skips one dataset, because bibliographic data sits outside its tuned domain. That's recorded as a skip. Scoring it zero would have been free points and also a lie.

Splink finishes faster, and the reason isn't speed

Someone asked me whether the wall-clock comparison was fair, since GoldenMatch derives its config at runtime while Splink is handed one. So I re-ran historical_50k locally, both engines on the same laptop, both on current main, and pulled the per-phase timings rather than the summary row.

Auto-config was never the cost. GoldenMatch's total was 97.41s: 94.02s matching, 0.63s loading, about 2.8s deriving the config. The published number already excluded that 2.8s, and excluding it changes nothing.

Excluding training from both sides makes the gap worse, not better. Splink's 6.70s is 4.67s of EM training, 1.59s predict, 0.44s clustering. Match against match and it's 94.02s to 2.03s. If you want the framing most flattering to me, it's the one already published.

The engines weren't doing the same amount of work. GoldenMatch scored 794,786 candidate pairs. Splink scored 152,897.

Comparing each engine end to end, including its own training:

Match-only against match-only, the per-pair figure is 8.91x rather than 2.78x. I'm quoting the end-to-end number because it's what a user experiences, and stating the other one because leaving it out is how the first version of this post went wrong.

So the wall gap decomposes into a blocking decision and an engine-speed difference, and the blocking decision is the larger term.

That wider net isn't an accident and it's where the accuracy comes from. Recall is 0.7474 against 0.6226. Pairs that Splink's tighter blocking never generates are pairs it can't recover, and on a dataset where no single field clears 0.60 recall, generating more candidates is how you find matches.

The accurate sentence: GoldenMatch is a few times slower per candidate pair, deliberately scores about five times more of them, and spends that time buying 0.125 more recall and 0.074 more F1. Whether that trade is worth it depends on what a missed match costs you. If you need an answer in seconds on fifty thousand rows, take Splink.

At scale the ordering reverses

On 50M rows and 463,923,179 candidate pairs, both arms on the same five-node cluster, Splink configured the way its own performance guide prescribes, GoldenMatch finishes in 552s against Splink's 1,054s.

The caveats matter here too. The margin narrows as size grows, from 2.54x at 1M to 1.91x at 50M. Single runs on this lane move about 16%, so no one ratio should carry weight. The fixture is synthetic.

Writing this post found two more

Both of these turned up while I was checking numbers for this piece, and neither is fixed yet.

The two harnesses measure different things and the docstring is wrong about it. run_splink.py documents its wall as "paralleling GoldenMatch's auto_configure+dedupe". It doesn't: GoldenMatch's timer starts after auto-config returns, while Splink's includes EM training. Correcting it makes GoldenMatch look worse, which is presumably why nobody noticed.

The harness knows the comparison isn't like-for-like, and I hadn't read the flag. Every result file carries comparable_to_splink, and in the published bake-off it's false. GoldenMatch's matchkey runs two ensemble scorers plus frequency-weighted and alias-aware name comparators; Splink's spec runs JaroWinkler, DamerauLevenshtein and exact match. There's a --fs-basic-scorers flag that rewrites GoldenMatch's scorers to match Splink's model, and with it wall drops from 94.02s to 64.92s. I'm not quoting that as a result, because the same flag also moves the link threshold, so it confounds three things at once.

Which means the per-pair number above is still not the last word. I'd rather say that than wait for a version of this post with no loose ends, because there isn't going to be one.

Why I wrote this down

A benchmark that only publishes its wins is marketing, and everyone can tell. The version of this post where I list three green numbers and stop is one nobody should believe, including me.

The other reason is that I've now hit the same failure twice in different domains. Earlier this year I fine-tuned a vision-language model to replace a paid extraction API, and before shipping it I checked the incumbent benchmark rather than the model. Its reported 0.926 accuracy turned out to be substantially the previous model graded against its own unreviewed output. Roughly 2,500 of 4,275 ground-truth records were the thing being tested.

Different stack, same shape. The measurement was wrong in the flattering direction, and nothing in the pipeline was going to tell me.

If you take one thing from this: when a benchmark agrees with you, that's the moment to go and read how it was computed.

Both write-ups, including every table above and the parts that make me look worse, are in the repo: github.com/benseverndev-oss/goldenmatch under docs/benchmarks/.