AI data quality: how do you clean and deduplicate data?

01Why customer data breaks AI answers

Google's machine learning course lists the usual causes of unreliable data: missing values, duplicate examples, bad values and wrong labels. Those problems hurt a trained model. They hurt an AI assistant even more directly, because the assistant repeats the bad value to a customer in fluent language. A placeholder price becomes a quoted price. A duplicate record becomes two recommendations for the same item. A wrong category sends a buyer to the wrong product. This is part of our larger guide, AI integration for SaaS: add AI without a rebuild.

Prompt work cannot fix this. You can tell a model never to invent prices, but if the feed says a unit costs one dollar, the model is not inventing anything. It is reading your data correctly. The fix has to happen before the data reaches the model.

02Profile every feed before you trust it

When customers supply the data, every tenant breaks the rules in its own way. Profiling each feed first shows you which rules it breaks. These are the problems we see most often, with the fix for each:

The important column is the last one. Most fixes belong in per-tenant configuration, not in code branches and not in the prompt.

03An AI data quality pipeline that holds up

We build the pipeline in this order, and we do not skip ahead to the AI step:

1. 01

   Profile each source

   counts, empty values, value ranges, duplicate keys and suspicious constants.

2. 02

   Normalize per tenant using configuration, so a new customer with a new quirk needs a mapping change, not a code fork.

3. 03

   Standardize vocabularies. Map categories, units and labels to one controlled list. In search engines such as Elasticsearch, a keyword normalizer applies the same cleanup both when data is indexed and

4. 04

   Resolve entities. Match records that refer to the same thing, merge them, and write survivorship rules that say which source wins when two systems disagree. Tools such as Splink do probabilistic linka

5. 05

   Link related entities, such as ingredients to recipes and recipes to menu items, so facts can be derived from the links instead of guessed.

6. 06

   Validate and report. Count rejected and corrected rows and report them back to the data owner, so they can fix the source.

7. 07

   Re-run on a schedule with change detection, because feeds change hourly and the fixes have to keep up.

04Where AI helps, and where it must not

AI is useful inside the cleaning pipeline. Language models can fill and standardize attributes such as company size or practice area. Similarity models can match a branded dish name to a generic one, or a long ingredient line to a clean ingredient. These are fuzzy language problems, and models handle them well.

Facts are a different matter. Prices, stock levels, nutrition values and account scores should come from linked, validated data, not from a model's guess. When nothing matches, the assistant should say so honestly rather than offer the closest thing as if it were an exact match. Keep a clear line between what the model may normalize and what it may never produce. For more on the gap between a working demo and production data, see why AI-generated code is not enough for brownfield SaaS.

05Exact matching does not scale

Comparing every record with every other record is too slow once you have millions of rows. The usual answer is to reduce the comparisons: turn names into numeric vectors, search with short word sequences, and use locality sensitive hashing so only likely matches are compared in full. The same blocking idea sits underneath probabilistic linkage tools. Plan for it early, because a matching step that works on a sample can stall on the full dataset.

06What we see in production

In a multi-tenant AI sales assistant for dealer websites, built in partnership with a dealership software company, each dealer's inventory feed broke the rules in its own way: placeholder prices of one dollar, missing prices, duplicate classification fields, RVs and boats mixed in one file, and different listing URL schemes.

We moved inventory from a CSV agent to search-engine-backed inventory search with filters, added inventory exclusion and new-versus-used rules, kept each dealer's data isolated, and built honest fallbacks for when no unit matches. Budget questions needed their own normalization too, since buyers think in monthly payments in the U.S. and biweekly payments in Canada. The assistant is live with dealers, and every answer is grounded in the dealer's own feed and website.

For a food discovery and nutrition startup, the data arrived as several disconnected JSON databases of restaurants, menu items, recipes and ingredients. We cleaned and normalized first: generic names for branded menu items, clean categories, ingredient lines split into quantity, unit, item and preparation, and duplicate ingredients merged. Then we linked them, using word2vec, word sequence search and locality sensitive hashing to make similarity search over the large set several times faster. Nutrition values come from the linked recipe and ingredient data, not from a model.

For a B2B SaaS company serving professional firms, one of our engineers works inside the client's own Azure and Microsoft Fabric environment, with its internal team, on a master database of customers and prospects. The approach we proposed puts survivorship rules first, entity resolution before anything else, and then a pipeline that uses a language model to fill and validate firm attributes, with scoring tuned to the client's own retention, expansion and deal velocity signals. Clean master data first, AI scoring second. The client owns the platform, and everything is documented so its team can run it.

In that master data build, the account, contact and activity layers were designed in Azure SQL first, and Microsoft Fabric came in only after entity resolution held, so the lakehouse starts from resolved records and the Ideal, Emerging, Acceptable and Avoid tiers go back into Salesforce on one merged account each.

Frequently asked questions