Data Matching Algorithms: Multi-Algorithm Record Linkage

Selecting, combining and evaluating string similarity and phonetic algorithms for entity matching. Data matching is often reduced to a deceptively simple problem: compare two values, calculate a similarity score and decide whether they represent the same entity. Production record linkage is considerably more complex. A person’s name, organisation name, postal address, telephone number and email address have different structures, distributions and failure modes. Typographical errors, character transpositions, abbreviations, token reordering, transliteration and phonetic variation all require different comparison strategies. The engineering challenge is therefore not simply: Which data matching algorithm is best?

A more useful question is:

Which comparison function should be applied to each attribute, under which conditions, and how should the resulting evidence be combined into a matching decision?

This is the principle behind multi-algorithm record linkage and the approach used by WinPure AdaptiveMatch™.  The resulting comparisons can be combined within predefined rules or weighted decision logic. Our guide to deterministic vs probabilistic matching explains how these approaches use field-level evidence to reach a final matching decision.

What Are Data Matching Algorithms?

Data matching algorithms are computational methods used to determine whether records from one or more datasets represent the same person, organisation, product or other real-world entity. They compare fields such as names, addresses, email addresses and phone numbers using techniques including exact matching, string similarity, phonetic encoding and token-based comparison. In practical record linkage, a single algorithm is rarely sufficient. Different fields contain different types of variation, so effective matching systems typically combine multiple algorithms, thresholds and field-level rules to determine whether records should be considered a match.

Common Data Matching Algorithms

Common data matching algorithms can be grouped by the type of variation they are designed to detect. Some measure character-level similarity, others compare tokens or word order, while phonetic algorithms identify values that sound alike despite being spelled differently.

No single algorithm performs best for every field or dataset. Effective record linkage therefore often combines several algorithms, with the comparison method selected according to the type and characteristics of each field.

Which Data Matching Algorithm Should You Use?

The appropriate data matching algorithm depends on the type of data being compared and the variations likely to occur. Names, addresses, email addresses and identifiers behave differently, so applying the same comparison method to every field can produce poor matching results.

These are starting points rather than universal rules. Algorithm selection should also consider the characteristics and quality of the actual dataset. In multi-field record linkage, the strongest results typically come from combining evidence from several fields rather than allowing one comparison to determine the match.

Record Linkage as a Classification Problem

Consider two records, Record A and Record B. The objective of a data matching system is to determine whether they represent the same underlying person, organisation, product or other entity. At its simplest, this can be treated as a classification problem: the system evaluates the available evidence and determines whether the records should be classified as a match or a non-match. The decision is rarely based on a single value. Instead, individual attributes can be evaluated using different comparison functions. For example, a customer matching process might compare:

Name → string similarity or phonetic comparison
Date of birth → exact or tolerance-based comparison
Address → fuzzy or token-based comparison
Email → exact or string similarity
Phone → exact comparison after standardisation

These individual comparisons create a collection of evidence that can be used to determine whether the two records represent the same entity. Importantly, there is no requirement for every attribute to use the same comparison function. In fact, there are good reasons why they should not.

Why a Single Similarity Algorithm Is Usually Insufficient

Different attributes exhibit different types of variation.

Consider the following pairs:

No single similarity algorithm handles all of these variations equally well. A phonetic algorithm may recognise that Steven and Stephen sound similar, but it is not designed to handle reordered company-name tokens or telephone-number formatting. Likewise, an edit-distance algorithm can identify character-level differences between Richardson and Richrdson, but may be a poor choice for fields where word order, pronunciation or standardised formatting is more important. Effective record linkage therefore applies comparison methods according to the characteristics of each attribute rather than forcing every field through the same algorithm.

Character-Level Similarity

Edit-distance algorithms are among the most widely used techniques for approximate string matching.

Levenshtein Distance

Levenshtein distance measures the minimum number of single-character insertions, deletions and substitutions required to transform one string into another. For example:

Smith → Smyth = 1 substitution

Jon → John = 1 insertion

Smythe → Smyth = 1 deletion

The lower the Levenshtein distance, the fewer edits are required to transform one value into the other. A distance of 0 means the two strings are identical. This works well for many typographical variations but does not inherently understand the meaning or semantics of the strings being compared. It also illustrates an important distinction between distance and similarity. A raw edit distance of two characters means something very different for a four-character string than for a forty-character string. Implementations therefore commonly normalise the distance when a comparable similarity score is required.

Damerau-Levenshtein

Damerau-Levenshtein extends the edit model to include character transpositions. For example:

Brian → Brain

A transposition-aware algorithm can represent this differently from an algorithm that must model the change as multiple independent edits. The distinction matters when adjacent character reversals are common in the source data.

Jaro and Jaro-Winkler

Jaro similarity considers matching characters and transpositions within a defined matching window. Jaro-Winkler extends this by increasing similarity when strings share a common prefix. That property can make Jaro-Winkler particularly useful for person-name matching, where early characters often carry useful information. However, the same prefix weighting that benefits one attribute type may be undesirable for another. This is precisely why the question should not be “Is Jaro-Winkler better than Levenshtein?” without first defining the data and error model.

Smith-Waterman-Gotoh

Smith-Waterman approaches comparison as a local sequence-alignment problem. The Gotoh optimisation introduces affine gap penalties, distinguishing between opening and extending a gap. This makes it possible to model contiguous insertions or deletions differently from multiple unrelated character errors. The technique is computationally more sophisticated than simple edit distance, but may provide useful behaviour where local alignment is more informative than whole-string transformation.

WinPureFuzzy™

WinPureFuzzy™ is WinPure’s proprietary fuzzy data matching algorithm and forms one of the comparison methods available within AdaptiveMatch™. Its purpose is not to replace every other similarity function. Instead, WinPureFuzzy™ provides a general-purpose fuzzy comparison capability that can be applied to suitable attributes and incorporated into broader matching rules alongside exact, phonetic and other algorithmic comparisons. This distinction is important. A similarity algorithm produces evidence about two values. It does not necessarily determine whether two complete records represent the same entity. For example, a strong fuzzy similarity between two surnames may provide useful evidence. The matching decision can become substantially stronger when supported by additional agreement on date of birth, postcode, email address or other identifying attributes. AdaptiveMatch therefore treats algorithm selection as part of rule construction rather than assuming that one similarity score should determine entity identity.

Token-Based Comparison

Character-level algorithms can perform poorly when strings contain similar tokens in different positions. Consider:

Advanced Data Systems Ltd

and:

Data Systems Advanced Ltd

The semantic content is highly similar, but the character sequence has changed substantially. Token-based approaches can normalise or reorder tokens before performing the underlying comparison. This can be valuable for organisation names and other multi-token fields where ordering is inconsistent. However, token sorting also removes information. If token order itself is discriminative, normalising it can increase the probability of false matches. As with edit distance, token-based comparison is therefore not intrinsically “better.” It represents another error model that is appropriate for particular data characteristics.

String similarity and phonetic matching measure different forms of variation and can be combined as complementary evidence within a data matching rule.

Phonetic Matching Is a Different Class of Comparison

Phonetic algorithms address a fundamentally different matching problem. Rather than measuring character similarity directly, they encode strings according to an approximation of their pronunciation. Two differently spelled values can therefore produce the same or similar phonetic representation without being identical strings. For example:

Steven ↔ Stephen

The spellings differ, but their pronunciation is similar. Phonetic matching can therefore identify relationships that character-based similarity algorithms may treat differently. This can be particularly useful for person names and surnames whose spelling has changed across systems, languages or generations.

AdaptiveMatch™ supports multiple phonetic approaches, including:

• Soundex
• Double Metaphone
• Kölner Phonetik
• Daitch-Mokotoff
• NYSIIS
• Caverphone 2

These algorithms should not be treated as interchangeable. Soundex, for example, was designed around English-language surname patterns. Daitch-Mokotoff was developed to better handle Slavic and Yiddish surnames. Kölner Phonetik targets German names, while algorithms such as NYSIIS and Caverphone use different transformation rules and linguistic assumptions. Consequently, simply deciding to “use phonetic matching” is not a sufficiently precise matching strategy. The more useful question is: which phonetic algorithm is appropriate for the population and attribute being compared?

Combining String and Phonetic Evidence

A useful matching system does not necessarily have to choose between fuzzy and phonetic comparison. They can represent separate pieces of evidence.

Conceptually, a name comparison might produce:

string_similarity =
    WinPureFuzzy(recordA.Name, recordB.Name)
phonetic_match =
    DoubleMetaphone(recordA.Name) ==
    DoubleMetaphone(recordB.Name)
postcode_match =
    recordA.Postcode == recordB.Postcode

The final record-level decision can then depend on the combination of these signals. This also provides better control over false positives. Phonetic encodings intentionally compress the representation of names. Multiple distinct strings can therefore map to the same code. Treating phonetic equivalence as definitive identity can produce collisions.Supporting evidence from other attributes can make the resulting rule substantially more discriminative.

Moving from Algorithms to Matching Rules

In a multi-field matching configuration, each field can have its own comparison algorithm, matching mode, threshold and weighting.

For example:

This is fundamentally different from applying a single fuzzy-matching algorithm and threshold across every field in a dataset.

For example, a global rule such as: Fuzzy match all fields at 85% assumes that every attribute behaves in broadly the same way. In practice, a person name, date of birth, postcode and email address contain different types of information and different forms of variation.

A multi-algorithm matching rule instead reflects both the meaning of each attribute and the type of variation expected within it. This allows each field to contribute appropriate evidence to the overall matching decision rather than forcing fundamentally different data types through the same comparison method.

Adaptive Algorithm Selection

As matching systems become more configurable, another problem appears: configuration complexity. A dataset containing dozens of candidate matching attributes can create a large configuration space involving algorithms, thresholds, phonetic options, matching types and rule weighting. AdaptiveMatch™ addresses this complexity by analysing each selected field and using its semantic data type and data characteristics to recommend an appropriate matching configuration. Rather than applying the same algorithm across every field, AdaptiveMatch™ can identify whether a column represents data such as a person name, company name, address, email address, phone number, date or identifier. It can then use this information, together with characteristics of the actual data, to select an appropriate algorithm, match type, threshold and weighting. For example, a person-name field may benefit from a string-similarity or phonetic algorithm, while an email address or identifier may require much stricter comparison logic. The resulting configuration can therefore vary from column to column within the same matching rule. AdaptiveMatch™ provides these selections as an automatically generated starting point rather than removing user control. Users can review the recommended configuration and, where required, change algorithms, thresholds, weights and other matching options.

The underlying principle is straightforward: matching configuration should adapt to the data rather than requiring every dataset to conform to a fixed global matching strategy. Technical users can still explicitly configure algorithms where domain knowledge justifies a different decision. This is particularly important because automated selection should be treated as configuration assistance rather than an assertion that an algorithm can infer every organisation’s definition of identity.

Thresholds, Precision and Recall

What Is a Good Data Matching Threshold?

There is no universal similarity threshold that reliably identifies a match. A score such as 80%, 85% or 90% may perform well for one field and dataset but poorly for another. The appropriate threshold depends on the algorithm, data type, data quality and the consequences of false matches. For example, a threshold suitable for comparing person names may not be appropriate for company names, addresses or product descriptions. Rather than selecting an arbitrary similarity score, thresholds should be evaluated against representative data and adjusted according to the required balance between precision and recall. Threshold selection introduces another optimisation problem. For a labelled matching dataset:

Precision = True Positives ÷ (True Positives + False Positives)

and:

Recall = True Positives ÷ (True Positives + False Negatives)

Reducing a similarity threshold will typically increase the number of candidate matches accepted. This may improve recall while increasing false positives. Increasing the threshold may improve precision while introducing additional false negatives. The optimum depends on the application. In some systems, incorrectly merging two independent entities can be considerably more damaging than leaving a duplicate unresolved. Matching thresholds can therefore be treated as a cost-sensitive decision. The appropriate threshold should reflect the relative consequences of false positives and false negatives. For example, in a compliance or identity-resolution scenario, incorrectly merging two different people may carry a much greater risk than failing to identify a potential duplicate. In other applications, such as marketing database deduplication, missing duplicates may be the greater concern. This is why matching thresholds should be validated against the intended use case rather than selected as a universal percentage.

Matching at Scale

Algorithm accuracy is only one dimension of a production matching system. A naïve all-pairs comparison across n records requires: Number of comparisons = n × (n − 1) ÷ 2

For one million records, this represents: 499,999,500,000 possible record pairs

Executing multiple sophisticated similarity algorithms against nearly 500 billion record pairs is clearly impractical. Production architectures therefore need to consider candidate generation, indexing, blocking or equivalent search-space reduction techniques before expensive comparison functions are executed. Algorithm selection also has computational consequences. A simple exact comparison, edit-distance calculation, phonetic transformation and local sequence alignment do not have identical CPU or memory characteristics. A matching strategy therefore needs to optimise not only statistical quality but also computational efficiency.

Benchmarking Matching Algorithms Properly

Algorithm comparisons based on a handful of hand-selected examples provide very little evidence about how a matching method will perform on production data. A more meaningful evaluation requires a representative labelled dataset containing known matches and non-matches. A benchmarking process should ideally include:

Ground Truth
↓
Candidate Generation
↓
Comparison Algorithms
↓
Threshold Testing
↓
Classification
↓
True Positives / False Positives / True Negatives / False Negatives
↓
Precision / Recall
↓
Runtime and Resource Measurement

Testing thresholds is particularly important. The same algorithm can produce very different precision and recall depending on where the acceptance threshold is set. Comparing algorithms at a single arbitrary threshold can therefore produce misleading conclusions. A useful benchmark should measure both individual algorithms and complete matching strategies. For example, this could include Levenshtein, Jaro-Winkler, WinPureFuzzy™, phonetic algorithms, multi-algorithm rules and an AdaptiveMatch™ configuration.

The evaluation should consider:

• Precision — how many accepted matches are correct
• Recall — how many known matches are successfully identified
• False positives — different entities incorrectly classified as matches
• False negatives — genuine matches that were missed
• Runtime — how long the matching process takes
• Resource usage — the computational resources required to perform the comparison

The objective is not to prove that one algorithm universally wins. It is to determine which configuration produces the required balance of matching accuracy, false-match risk and computational performance for a particular dataset and use case.

The Engineering Case for Adaptive Matching

There is unlikely to be a universally optimal string-matching algorithm because record linkage is not a single similarity problem. It is a collection of comparison problems operating across attributes with different semantics, distributions and error characteristics. Levenshtein can model character edits. Damerau-Levenshtein can account for transpositions. Jaro-Winkler can emphasise common prefixes. Token-based methods can address reordered terms. Phonetic algorithms can identify pronunciation-related variation. WinPureFuzzy™ provides WinPure’s proprietary fuzzy comparison capability. The more important engineering question is how these techniques should be selected and combined. AdaptiveMatch™ is designed around this principle.

Instead of assuming that every column should use the same algorithm, matching mode and threshold, AdaptiveMatch allows comparison strategies to be selected at the attribute level and combined into more expressive matching rules. Where appropriate, it can automatically determine suitable configurations based on the characteristics and semantic type of the data while retaining explicit control for technical users.

The result is a shift away from asking:

“Which fuzzy matching algorithm should we use?”

towards the more useful question:

“What combination of comparison functions provides the strongest evidence that these two records represent the same entity?”

For production record linkage, the goal is not to find a single “best” algorithm, but to combine the right comparison methods for the data, attributes and matching objective.

Frequently Asked Questions

Written by

Team WinPure

The WinPure Team brings together over 100 years of combined expertise in data quality, data management, data matching, entity resolution and software innovation. Our growing team of experienced specialists combines deep industry knowledge with a passion for advancing data quality technology, helping organisations worldwide improve data accuracy, identify and resolve duplicate records, streamline operations and unlock the full potential of their data.