An Overview of Data Matching

Simply defined, data matching is a comparison process. You compare data attributes or values to determine if they are similar to each other. On an advanced level, data matching compares large data sets to determine if multiple attributes are tied back to one entity. If an entity has attributes stored in multiple data sets, data matching allows analysts to match and combine these attributes to get a unified view of the individual.

Real-world entities are recorded and identified in databases based on attribute values – which are descriptions such as names, addresses, phone numbers, and so on. When recording this data, either manually or via automatic means, there is always a chance of error.

For example, a data entry officer may accidentally write a name as Catherine instead of Kathryn, a variation of the same name but with different spellings. Moreover, the same person can have multiple email addresses, phone numbers, and physical addresses. It’s not uncommon for people to sign up to products or services with multiple emails. These data duplication issues eventually cause the database of an organization to “bloat,” resulting in flawed analytics and insights. 

Imagine your marketing department is drafting a yearly progress report. If your CRM is bloated with 10 – 15% duplicate data, the report you’d get would be unreliable! You’d think the organization has an increase of 20% in customer acquisition, while in reality, the figure is way less than this. This is where a data matching solution is needed to help your teams treat their data for errors, duplicates, and disparity.

Data duplication is just one of the many use cases of data matching. Other use cases include:

• Entity resolution: determining and linking different data entries that refer to the same real-world entity.
• Identity resolution: verifying and matching multiple attributes or identifiers to establish the true identity of an individual.
• Record linkage: linking information about one individual spread over multiple systems (such as a government benefits database)
• GDPR/sanctions: matching a company’s database with government databases to ensure sanctions and privacy law compliance.
• Customer360 view: enabling teams to get a consolidated view of their customer data across systems.

These use cases are the ‘dream goals’ of most organizations, however, most companies buckle under the weight of the processes involved in achieving these goals. Every year, companies spend millions of dollars in hiring and retaining skilled data scientists and analysts just so they could help them achieve these goals. Although it sounds fancy, the process itself involves a massive amount of redundant, mundane work that occupies hundreds of hours, leaving these skilled specialists stuck in non-strategic roles.

Let’s dig into this further and see how most companies today perform data matching to achieve the above goals.

The data matching process consists of five major steps, starting from cleaning and standardzing the data, to matching, and finally creating master or golden records. 

Here’s an example: 

There are two database tables that are to be matched.

Both contain the same name, address, DOBs and phone numbers. The problem? They differ in terms of structure and content quality. Moreover, they have different unique IDs. This data needs to be consolidated so it can become a reliable source. 

The following process is used to match Dataset A and B.

 Source: Data Matching Process by Peter Christen

Most data sources contain incomplete, incorrect, and inaccurate data that affect the accuracy of a matching process. Before initiating a data matching process, you must clean, format, and reduce errors to avoid a false match. There are real-world consequences to incorrectly matched data, such as a bank mistakenly sending sensitive information to the wrong individual, leading to court cases, penalties, or violation of privacy laws.

Data cleaning, therefore, is the most critical step of data matching.

There are various causes of incorrect or dirty data attributes – from manual data entry (such as someone typing in data), to form fields on websites that do not have data quality checks in place (such as automated country code numbers to prevent incorrect manual entries) as shown in the image here: 

There are generally four types of data cleanups to focus on: 

a). Removing stop words and irrelevant characters: This step aims to clean the input data so that only relevant information remains. For example, commas and periods are removed because they can clutter an analysis; likewise, with semicolons ( ; ), quotes” terminate sentences. In applications like WinPure, you can also remove common stop words such as articles, prepositions, pronouns, etc. (or, and, but the), which do not add any value to the information in the text.

 b). Fixing spellings & abbreviations. Along with typos, spelling differences are significant issues in data matching. For instance, Johnny is a common nickname for John. If you’ve collected John’s data from a third-party app such as Facebook, where he uses Johnny, you’ve got a duplicate & an inconsistent entry.

This process often relies on look-up tables containing name variations, nicknames or common variant spellings—along with their expanded versions–to ensure a standard set for all values throughout your system. Standardization improves how well attributes match up because it reduces variation in names across different records and ensures consistency.

 c). Parsing fragmented attributes: This process deals with attributes that have fragmented pieces of information. For example, matching the street, postal code, and suburb columns of the first set with the text column of the second set is challenging. In such cases, you would need to parse the data through manual methods or a data matching solution that allows you to parse the data based on multiple probabilistic techniques automatically.

d). Verification of attribute values: The final step in the data cleaning process is to verify the correctness of addresses against available databases. Also called address verification, available addresses are matched against a reliable external source like the US or UK postal database, where locations can be verified. For example, it is known that there’s no town called, ‘ShallCruss” in a popular time, but there’s a “Shallcross Crescent,” and that will prompt a check during the process. Popular data cleaning solutions have address verification as an essential module of the software that lets users verify against government databases in the US, UK, and Canada.

Here’s how the above database looks after a data cleaning process.

Note: With WinPure, you can also append a column such as a Gender column for more accuracy. 

2. Preparing for Match with Indexing

Once the data is cleaned and standardized, each record from one database will be compared with all other databases to calculate similarities. While this seems straightforward, it’s tough to manage with large databases. 

For example, matching two databases with 1 million records each will result in 1 trillion pair comparisons. Such comparisons are computationally heavy and complex and are not feasible for today’s large databases that may contain millions of records. The number of possible matches *and* non-matches for these records would be exponentially huge, making the data matching process unnecessarily complex.

Indexing helps resolve the “match all pairs” problem by reducing the number of record pairs that need a comparison. Without indexing, there would have been m × n detailed comparisons for every pair where only one was necessary if we assume likeliness between records equals 100%. 

Indexing techniques involve the creation of blocks, lists, or clusters. The key values on which these operations depend often come from one attribute but could include several different ones if needed.

For example, a postcode (or zip code) attribute could be used to block all records that have the same value. This would make it easier for users looking up specific addresses by assigning them one index list with blocks of data. Indexing is important for more than just matching data. It needs to be applied when deduplicating a single database as well. The techniques used during indexing can also help with matching any new databases that come across yours – saving time on repetitive tasks.

3. Record Pair Comparison

Indexing may generate potential matches, however, more details are required to define a true match. That’s when record pair comparison comes in. You attempt to find two records that share as many similarities as possible. 

Generally, the greater the number of attributes shared by two given individuals will result in a higher degree of matching between them. Using the running example, we can find more potential matches by including postal codes and phone number columns to narrow the match.

The ability to find a match between two sets of data is only the first step in addressing incomplete matches. Even after all possible characters have been matched and standardized, there may still exist different attribute values that correspond with true matches. In this case, ‘100’ and ‘106’ likely refer to the same person, but their respective First Names differ just enough such that they would not be considered exact duplicates. Therefore, it is necessary to conduct some approximate comparison to measure similarity.

Different attributes contain various types of data, and this requires different comparison functions to be applied. For example names with string values such as “name” or addresses that consist only of letters cannot be compared using the same criteria as dates which also include numbers in some cases; thus it is advised you compare groups when dealing with these sorts of attributes.

4. Classifying as a Match or a Non-Match

In traditional data matching approaches, for example those based on probabilistic record linkage models are classified into one of three classes: (i) matches (ii) non-matches, or (iii), undecided cases.

The difference between a match and non-match is based on how the records are related. A match is when the pairs of records refer to the same real-world entity. A none match is when the two records do not refer to the same entity. The third classification is a potential match that calls for a manual review.

5. Ensuring the Data Quality of the Matches 

Match quality is an essential measure of how well matches are identified and is greatly affected by how the data cleaning was performed. If you miss cleaning unnecessary characters (such as hyphens between phone numbers), you could be looking at inaccurate and false match results.  

Remember, though, much of the assessment of quality matches requires human intervention – from carrying out extensive investigations to double checking individual records or contacting them to determine the truth about the match. With a powerful data-matching solution, data analysts can spend their time where it is needed instead of being stuck in the endless loop of testing and tweaking match algorithms. 

While there are multiple algorithms and methods to data matching, most data match algorithms are built on the foundation of three types of match conditions. These are: 

a. Fuzzy Matching 

Fuzzy matching allows for easy matching of semi-structured data and records that do not have exact matching attributes. Text strings like names are matched using fuzzy techniques such as Soundex for same sounding names, or Levenshtein Edit Distance for differences in spellings. 

For example, the edit distance between the strings Catherine and Katherine is “1” because only one edit operation, the substitution of C for K is necessary to transform Catherine into Katherine.

The significant drawback of using fuzzy data matching is the potential for false positives and false negatives. Fuzzy matching algorithms introduce a level of uncertainty and subjectivity into the matching process, as they rely on similarity measures and probabilistic methods. This can lead to incorrect matches where different entities are mistakenly identified as the same, or genuine matches are overlooked due to insufficient similarity.

Inaccurate matches can have serious consequences in various applications, such as customer relationship management or fraud detection, leading to incorrect decisions, wasted resources, and damaged relationships. Therefore, careful consideration and validation are necessary when employing fuzzy data matching to ensure the reliability and accuracy of the results.

b. Exact Matching 

In this technique, you match text that you are 100% sure is correct or exact and doesn’t require any additional edits. 

For example, after you’ve treated spelling inconsistencies and want to match everyone named Catherine to identify duplicates, an exact match is the easiest method to go with. 

However, a problematic limitation of exact matching is its inability to handle data inconsistencies or variations. Since exact matching relies on strict criteria of identical values, even minor differences or errors can lead to missed matches. For example, a typographical error, a slight variation in formatting, or the use of abbreviations can result in failed matches. This can be particularly problematic in real-world scenarios where data quality may be compromised or when dealing with large datasets.

The lack of flexibility in accommodating small variations makes exact matching less robust and can hinder its effectiveness in scenarios where some degree of inconsistency is expected.

c. Numeric Matching

Numeric matching deals only with numbers. It’s great for matching phone numbers or postal codes that contains only numbers. 

Similar to exact matching, numeric data matching has precision issues. It relies heavily on the accuracy and consistency of numeric values. However, when dealing with large datasets or complex calculations, rounding errors or inconsistencies in decimal places can occur. These small discrepancies can lead to mismatches or inaccurate results.

Data analysts would have to spend weeks tweaking, testing, and coding algorithms based on these match approaches to resolve problems with text, string, and numeric data. An automated solution however can perform matches using a combination of these algorithms within minutes, allowing the users to adjust match flexibilities and thresholds for multiple data sets at a time.

Data matching solutions allow you to pre-process, index, block, and compare within and across multiple data sets in an easy on/off premises platform. With WinPure for example, you can match data in five easy steps: 

• Integrate data sources from multiple data sets & file formats: Unlike a few decades ago, you no longer need to manually transform data to run a comparison. With easy integration functions, you can connect a CSV file or a MySQL file to the interface and begin a match process. 

• Advanced data cleaning functions: In the image given below, you will see how the tool profiles the data for inconsistencies and errors. So if you’re in the marketing department, you can see straight away you’ve got empty email addresses and fields with punctuations and characters that add “noise” to the data. Traditionally, it would take hours and weeks to clean this data, because if it’s not clean, it leads to unreliable insights that directly impact business and revenue. 

• Advance cleaning with custom regex expressions: Sometimes you’ve got complex string data such as email ids that contain numbers and text such as [winpure123@winpure.com]. You can match these strings using advanced regex expressions that are built in the tool or that you can get from a library of expressions to match complex data.

• Standardizing and cleaning data with splitting the data: When you have mutliple data sets, such as from market, product management, or sales, you can end up with inconsistencies in standards. For example, someone can write the data structure as dd//mm//yyyy and someone can write it as dd/mm/yy. 

This supposedly small discrepancy can cause discrepancies in match results. It’s always recommended to standardize problems such as capital letters, spaces between code numbers, abbreviations and others.

You can resolve these issues on the WinPure platform by splitting the data and choosing options like Propercase, Uppercase etc to identify and resolve standardization problems. 

• Building your own word library: Have specific words and abbreviations that you want to consider during the match process? WinPure lets you build a custom word library using Word Manager, a function that lets you build your own records about how you want your data to be corrected or standardized. For example, you prefer Limited over LTD, or Ltd. Whichever your preference, you can set the standard and the tool will correct it accordingly. 

• Matching within and across data sets: From the columns you’ve cleaned before, you can now match within data sets (such as matching data of Table A, then Table B). Once done, you can then match data across the tables (A x B) to weed out duplicates.

• What to match? Obviously the data sets that you believe share unique traits. For example, matching Contact Numbers with Company Names. Two contact numbers for company A is a duplication. When choosing what to match, it’s better not to overcomplicate the decision and opt for attributes that you know have lower chances of similarity. For instance there could be multiple individuals with the name Johnny, but there’s only one company named Apple!

• How to match? Fuzzy, exact, or numeric? Everyone does this section differently. You can select 90% fuzzy match to see how the results turn out or you can do an exact match for data you already know has been fixed and standardized. You can also do numeric matches for phone numbers and postal codes and exact matches for data that has been thoroughly processed. 

9. Assessing the match or creating master records: Once you get the match results, you can decide whether you want to manually assess the results (evaluation) or you’re reading to save it as a master record. Remember, master records become the most reliable version of the data, so always make sure your data is processed and deduped before you create a master record for use later on.

…. And there you go! Your match done! Using an automated data matching solution can save you at least 4 weeks of manual work and improve efficiency by 70%! 

When done right, data mataching can help firms and government organizations achieve key objectives like: 

1. Fraud detection and prevention

Financial institutions are under immense pressure to deal with increasingly complex fraudulent activities. From scams to fake identities, from money laundering to punitive regulatory compliances, firms need access to accurate, reliable, consolidated records. Data matching is used to compare the firm’s records against criminal and sanctions databases to identify details about the individual.  

2. Better Public Programs

A CBPP report in the United States, revealed over 40% of eligible individuals missed out on a public nutritional program for women, infants and children because of enrollment gaps that prevented the individuals from actually getting the benefits. Data matching allowed four states to find enrollment gaps, and also to identify individuals for targeted outreach. Additionally, the data matching also reduced the applicant’s documentation burden and simplified the certification process for eligible individuals. Data matching has repeatedly been demonstrated as a necessary and beneficial technology to uplift, improve, and deliver effective public programs.

3. Targeted Campaigns

According to Salesforce 70% of CRM data becomes obsolete while around 30% of records are duplicated. Yet companies will still send out emails, direct mails, flyers to all existing database customers only to be at the receiving end of the customer’s wrath. Data matching becomes a necessary activity. 

Companies would need to regularly de-deduplicate records and ensure they are complete, updated, and accurate whenever they want to run a campaign. Most importantly, data matching helps with identifying household data (members of a family using the same service), which in turn enables marketing teams to build targeted lists. For example, an insurance company can cut down on the costs of direct mailing by up to 80% if they send out just one flyer to one household instead of five flyers to five members of the same household!

4. Improved Customer Service

Data matching can be used to consolidate scattered customer records giving companies a 360 view of their customer journey. This in turn helps marketing and customer service departments improve customer service. For example, an airline can use data matching to identify customers’ preferences and provide them with optional stays at exclusive hotels or BnBs.  

5. Improved Customer Retention

For example, a gym can use data matching to offer exclusive discounts to members of the same family. Similarly, an insurance company can use data matching to offer specific health insurance plans for people with newborns or people with teenagers. This ability to predict customer needs and offer a relevant service or product greatly boosts customer retention.

6.  Increased Organizational Efficiency

When teams have access to accurate and reliable data, they can make decisions faster and better. Companies that have invested in MDM and entity resolution processes have reported higher efficiency of up to 80%! 

7. Remove Duplicates & Improve Data Quality

One of the biggest benefits of data matching is deduplication – the process of removing duplicates within a data source. Duplicate data is a critical data quality concern that most firms are ill-equipped to handle because of informal data matching processes that result in higher false negatives and positives.

8. Drive Business Growth

Efficient data matching is the backbone of entity resolution which boosts growth factors like complete customer views, targeted marketing, better product and services and so on. If businesses want to compete in today’s world, they need reliable data systems and processes.

Most firms we’ve worked with are well aware of the benefits of data matching but struggle with the implementation process. Our Clean and Match solution is designed to take the struggle away and help you and your team breeze through a complicated process with a point-and-click, user-friendly, no-code solution that ensures you meet your business goals fast.

Data matching can be done manually, and most companies do employ expensive data analysts to use Excels for data matching – but – as datasets become larger and more complex, it is practically a nightmare to manual match and clean data. Ask any of your data engineers and they would likely call data matching one of the most mundane and toughest part of the job. 

Some of the key data matching challenges that happens when you try to do it manually involves: 

Variability in data: The fancy term for this is ‘data heterogeneity,’ which describes variability in a data caused by factors such as differences in the data’s source, structure, or format. Take for example a customer database may include customer name, address, and contact information, while a product database may include product name, SKU, and price. To match these two databases, the data must be standardized so that the same type of information is included in each record. This can be a time-consuming process and often requires human intervention.

Preventing data matching errors: False positives and negatives are two types of errors make data matching a challenging process. False positives – when a record is erroneously matched to the same master record, and false negatives – when a record is not matched even though they belong to the same entity record. 

Privacy and the lack of data: For most data matching processes, it is almost impossible to obtain data due to privacy laws. There are numerous studies on the best way to access and solve data matching problems based on privacy restrictions, and one of the most popular models is proposed by experts is the One-way hash encoding functions to convert a string value into a hash-code (for example ‘peter’ into ‘51dc3dc01ea0’) such that having access to only a hash-code will make it nearly impossible with current computing technology to learn its original string value. There are many other models including Secure multi-party computation (SMC), phonetic encoding, bloom filters and many more. 

Data matching is commonly used in financial services, healthcare, and retail. In financial services, data matching is used to identify duplicate accounts, fraud, and other money-related crimes. In healthcare, data matching is used to identify patients who have been seen by multiple doctors or received multiple prescriptions from different pharmacies. In retail, data matching is used to identify customers who have purchased the same product from multiple stores.

Financial Services

The goal with data matching in the financial service industry is to identify potential matches between customers and banks in order to assess risk and prevent fraud.

One example of data matching in financial services is when a bank reviews an application for a new credit card. The bank will compare the customer’s information (name, address, etc.) with its own records to see if the customer already has an account with the bank. If they do, the bank will likely not approve the new credit card application, as it would be considered high risk.

Another example is when a bank detects fraudulent activity on one of its accounts. The bank will compare the account information (account number, name on account, etc.) with its own records as well as with records from other banks. This helps the bank to identify any potential patterns in fraudulent activity and better protect its customers.

Data matching is also used for compliance purposes in financial services. For example, banks are required to keep track of all of their customers’ transactions over a certain period of time. This information is then compared against regulatory guidelines to ensure that the bank is in compliance.

The benefits of data matching for financial services include improved risk management, decreased fraud losses, and increased compliance efficiency. Data matching helps to identify potential risks before they become problems, which can save the bank money in terms of losses and fines. It also helps to streamline compliance processes by automating the comparison of data against regulatory guidelines.

Government and Public Sector

Let’s consider the example of a city government that wants to improve its decision making around budgeting. The city government has two datasets: one that contains information about the current budget and one that contains information about past budgets. The city government wants to use data matching to identify discrepancies between the two datasets.

By identifying these errors, the city government can identify accounts that have been tampered with or where numbers didn’t add up. Additionally, the matching process may also help with different information about one entity (multiple versions of a name).

Education Industry

In the education sector, data matching can be used to match data from different sources, such as student information systems and alumni databases for multiple purposes – from monitoring student progress to tracking benefits, to ensuring performance benchmarks, to providing mental health support on time.

In some cases data matching can help with identifying outdated data. For example, if a student changes their name or moves to a new school, the data in different systems may not be updated with the latest information.

Healthcare Sector

Data matching is a pivotal function in the healthcare sector where patient records such as their medical history, treatment history, lab results are always matched and consolidated to provide their specialists with an overview of their health and the treatment they require. It is not uncommon for patients in the healthcare sector to suffer lethal and often times tragic consequences of mismatched IDs (such as providing a wrong medication or perform a misdiagnosis).

The use case below describes how healthcare sector organizations can use data matching to improve patient safety and care.

A patient is admitted to the hospital with a suspected infection. The hospital staff enter the patient’s information into the hospital’s electronic health record (EHR) system. The EHR system automatically compares the patient’s information against a database of patients with known infections. If the EHR system detects a potential match, the system notifies the hospital staff so that they can take appropriate action.

This use case illustrates how data matching can be used to improve patient safety and care in the healthcare sector. By automatically comparing patients’ information against a database of patients with known infections, hospitals can identify potential matches and take appropriate action. This helps to ensure that patients receive the correct treatment and avoid potential complications.

Marketing and Sales

In the marketing and sales domain, data matching is used to perform a plethora of activities – from identifying potential customers, to finding common interests among customers, to running targeted marketing campaigns and much more. In fact, of all the industries, marketing and sales benefits the most from data matching especially since it deals directly with customer data.

Takek for example, a company that wants to improve its marketing and sales efforts by matching customer data with potential new customers. It has a database of current customers and would like to find potential customers based on the attributes of the current ones. The first step is to gather and assess data like customer names, addresses, and phone numbers, such as their purchase history, reviews, and other relevant information. Once this data is gathered, it is cleaned and standardized and a matching algorithm is created to compare the data from both databases. Matching results provide insights on customers of certain areas that are more receptive to marketing offers. Based on this info, a targeted marketing campaigns or to find potential new customers. The data could also be used to improve sales efforts by identifying potential leads for sales representatives.

Data Matching is a critical function that demands Best-in-Class technology.

As data structures become more complicated, it requires data matching technologies that can keep up. Whether it’s an enterprise project like matching multi-million records for a merger, or a small departmental project like combining marketing & sales data for insights, primitive data matching techniques such as exact comparisons via Excel formulas or manually coded algorithms are no longer enough or effective. Organizations need best-in-class data matching solutions that offer data preparation, data cleaning, and data transformation as part of the process. Moreover, an intelligent data matching technology must have multiple algorithms at work to tackle data nuances and reduce errors.

What is data matching used for?

Identifying duplicate records, verifying the accuracy of data, and consolidating data. 

What is matching in data quality?

Identifying and correcting inconsistencies between data sets. 

What is an example of data matching?

Comparing two records to identify if they are duplicates. 

What are the types of matching?

There are three types of matching; fuzzy, exact, and numeric among many others. 

What are data matching issues?

Incorrect or incomplete data, mismatches in data formats, and differences in coding schemes are common data matching issues.