The engineering team at Hightouch published experimental results showing that a hybrid pipeline combining fast decision models with frontier LLMs can audit Identity Resolution (IDR) outputs at frontier-level accuracy while cutting costs by more than 4x.
The Problem: Detecting Over-merging in IDR
Identity Resolution maps disparate data records to real-world individuals. A critical failure mode in IDR is over-merging, where records belonging to multiple distinct people are mistakenly grouped into a single identity profile. Because real-world production data lacks an answer key, engineering teams typically rely on manual human review to estimate error rates. Human sampling is slow (roughly 50 identities per hour), and narrowing the over-merge rate estimation error to under ±1% requires inspecting over 1,000 samples.
To test whether LLMs can automate this auditing process, Hightouch generated a synthetic benchmark of 1,000 identities using pseudopeople, a simulation tool from the University of Washington. Half of the profiles represented a single person with realistic data noise (typos, swapped birth date months and days, nicknames, and household value copying), and half represented true over-merges.
Comparing Frontier LLMs and Decision Models
The team evaluated frontier models (Claude Opus 5.5, Gemini 3.1 Pro, and GPT-6 Sol) alongside a newer class of decision models—specifically Jev and GPT-6 Luna—which output a 0–1 confidence score indicating whether a condition holds. Out of the box, decision models are up to 90x cheaper ($0.05 to $0.10 per 1,000 identities) and over 10x faster (~0.15 seconds per identity), but trail Opus accuracy by roughly 6 percentage points on harder benchmarks.
| Model / Setup | Accuracy (Easier Test) | Cost / 1,000 (Easier) | Accuracy (Harder Test) | Cost / 1,000 (Harder) | Time / Identity (Easier / Harder) |
|---|---|---|---|---|---|
| Claude Opus 5.5 alone | 99.0% | $3.86 | 95.1% | $5.39 | 2.2s / 2.6s |
| Jev alone (60% cutoff) | 95.7% | $0.05 | 89.1% | $0.06 | 0.15s / 0.15s |
| GPT-6 Luna alone (70% cutoff) | 97.3% | $0.10 | 88.8% | $0.12 | 0.17s / 0.20s |
| Hybrid Jev (30–90% to Opus) | 98.8% | $0.88 | 95.3% | $2.94 | 0.63s / 1.56s |
| Hybrid Luna (10–90% to Opus) | 98.5% | $0.33 | 91.6% | $1.15 | 0.30s / 0.70s |
Why This Matters for Engineers
For engineers designing LLM-based evaluation harnesses and routing pipelines, the experiment highlights several practical system-design takeaways:
- Confidence calibration matters more than raw standalone accuracy: Out of the box on the easier dataset, GPT-6 Luna outperformed Jev (97.3% vs. 95.7%). However, Luna's errors were heavily concentrated in the 90–100% confidence bucket. Jev's errors clustered closer to 50% confidence, demonstrating much better calibration.
- Self-scaling hybrid architectures: By routing uncertain predictions from Jev (confidence between 30% and 90%, about 22% of the easier dataset) to Claude Opus 5.5 for a second opinion, the hybrid system reached 98.8% accuracy at $0.88 per 1,000 identities—over 4x cheaper than running Opus alone ($3.86).
- Resilience on adversarial/hard data: When tested on a harder dataset featuring twins, junior/senior name collisions, and severe typos on key fields, Hybrid Jev automatically routed more cases to Opus without any prompt or threshold changes. It achieved 95.3% accuracy at $2.94 per 1,000 identities (matching Opus alone at 95.1% and $5.39). Conversely, Luna's severe overconfidence prevented it from escalating tricky cases, causing Hybrid Luna's accuracy to drop to 91.6%.
- Prompting behavior differs by model class: Including explicit examples of data entry errors improved accuracy across all models. However, explaining the underlying data sources (census, tax forms, etc.) only added up to 1 percentage point for frontier models and did not help decision models at all, suggesting decision models do not perform deep contextual reasoning.
