Fixing Duplicate and Dirty Customer Data in Dynamics 365 Without Breaking Reports

Why do duplicate records keep coming back in Dynamics 365?

Because the platform never promised to stop them. Duplicate detection rules find likely duplicates and warn, and the person saving can still save. They only run where they are switched on, and each ingestion route behaves differently. Until an alternate key exists, nothing in Dataverse enforces that a customer can exist only once. A one-off cleanup that does not change the table below will be repeated.

Which duplicate records should you clean up first?

The ones carrying money. Duplicate accounts with open opportunities, contracts, orders or invoices split pipeline and revenue across two names, which is the version of the problem that reaches finance and auditors. Next come the contacts on those accounts, because they drive who receives quotes, invoices and service. Leads and dormant contacts are the long tail: real, but rarely the reason a report is wrong.

Prioritising this way also keeps the first batches small and well understood, which is where the merge method below needs to prove itself before it touches volume.

How do you profile and quantify duplicates before merging anything?

Profile outside the live system, on an export, where you can normalise freely. Lowercase and trim emails, strip punctuation and legal suffixes from company names, normalise phone numbers to one format, extract the domain from emails and websites, and use registration or tax numbers wherever they exist. Then group candidates by the strength of the evidence: an identical registration number is not the same kind of match as a similar name in the same city.

The output is a number per table and per match tier, and, for each candidate group, what is attached to it. That is the figure that turns a cleanup into a funded project, and it is the only defence against the most expensive mistake in this work: merging two genuinely different customers whose names look alike. If the data arrived through a migration, the load itself may be the source; our Salesforce to Dynamics 365 data migration guide covers how source keys and case sensitivity create duplicates during a load. Where two CRMs are being consolidated after an acquisition, the match tiers and survivorship rules are set out in our post-merger consolidation guide.

How should duplicate detection rules be tuned to your real data?

Build rules from what profiling showed is reliable, not from the defaults. A rule on company name alone flags every branch of a franchise; a rule on email alone misses people who use two addresses. Useful habits:

• Match on the columns that actually identify a customer in your data, such as registration number, normalised email or website domain, and combine a weak column like name with a stronger one.
• Use the ignore blank values option, otherwise every record with an empty column matches every other.
• Use exact match or same first characters deliberately; a partial match on short values produces noise that users learn to click through.
• Keep the number of published rules small and meaningful, because each one adds a check to every save and a warning people will stop reading if it is wrong too often.
• Run a duplicate detection job with the rules and compare what it finds with the profiling results before switching the rules on for users.
• Remember that rules warn. For values that must be unique, the enforcement comes from an alternate key.

What does merging do to activities, cases and financial history?

Merge is available for accounts, contacts, leads and cases. You pick the surviving record and choose, column by column, which values it keeps. The platform then moves related child records onto the survivor and deactivates the other record (a merged case is cancelled), marking it as merged and pointing it at the survivor. It is not deleted, but there is no unmerge.

How do you merge in reversible batches when there is no unmerge?

You make merges reversible yourself, before you run them. For every batch, record the surviving record ID, the subordinate record ID, the column values that were overwritten and the IDs of every child record that will move. With that log, a wrong merge can be undone by reactivating the subordinate, restoring its values and reparenting its original children back to it. Without it, a false positive merge between two real customers mixes their revenue and service history permanently, and nobody can reliably separate it again.

Keep batches small enough that the reports can be reconciled between them, start with the strongest matches, and route weaker matches to a review list owned by the people who know the customers. Automating merges at scale through the API is possible, but only after the manual batches have proven the rules and the log.

• Snapshot reports and related records before each batch: Save the key report figures and, for every record in the batch, the IDs of its related activities, cases, opportunities, orders and invoices, so the batch can be reconciled and reversed.
• Merge in small, logged batches: Merge the strongest matches first, record the survivor, the subordinate and the moved child records for every merge, and hold weaker matches for review by the people who know the customers.
• Reconcile reports after every batch: Compare the saved figures with the new ones and explain every difference before the next batch, including reports and exports that key on the subordinate record ID.

How do alternate keys stop new duplicates from being created?

An alternate key tells Dataverse that a column, or a combination of columns, must be unique in a table, and the platform enforces it with an index: a second record with the same key value is rejected whatever route it arrives through. It also lets integrations upsert, updating the matching record instead of creating another. This is the step that makes a cleanup sustainable.

Three practical points. The key cannot activate while duplicates on that column still exist, so it comes after the merge batches for the records it covers. Matching on a key is not case sensitive, so normalise the value into a dedicated column rather than keying on free text people type. And decide how records without the value, such as a contact with no email, are handled, testing that in a sandbox before activating the key in production. The same technique on migration source IDs is covered in the bulk import guide.

Which intake controls stop forms, imports, integrations and marketing sync creating duplicates?

Every route from the inventory needs its own control, because one unguarded route refills the table. Integrations are usually the largest source, and where there are many of them the underlying problem is the integration estate itself; our guide to rationalising point-to-point integrations covers that.

• Forms: duplicate detection on create and update, a search-first habit for account and contact creation, and required identifying columns only where people genuinely have the value.
• Imports: duplicate detection enabled in the import, and repeatable imports switched to upsert on the alternate key.
• Integrations: upsert on the alternate key, or requests that set SuppressDuplicateDetection to false and handle the resulting error deliberately.
• Marketing forms and sync: a matching setting reviewed alongside the rules in the app, so a form submission updates a known contact.
• Email tracking: review whether contacts are created automatically from tracked email, and by whom.
• Monitoring: a scheduled duplicate detection job, a view of new candidates and a named owner who clears it.

What is the step-by-step method for a sustainable duplicate cleanup?

This is the whole sequence in order. If you want the current state measured independently before committing to it, data integrity is one of the dimensions of our Dynamics 365 health check and technical audit.

• 1. Inventory every route that creates customer records. List forms, the import wizard, each integration, marketing forms and sync, email tracking that creates contacts automatically, and any bulk tools, and record for each whether duplicate detection or an upsert key applies to it.
• 2. Profile and quantify the duplicates. Export accounts, contacts and leads, normalise names, emails, phone numbers, domains and registration numbers outside the system, and count candidate duplicate groups by match strength, together with the open opportunities, cases, orders and invoices attached to each.
• 3. Prioritise the records that carry revenue. Start with duplicate accounts that hold open pipeline, contracts, orders or invoices, because they distort the numbers finance and management read, then contacts on those accounts, then the long tail.
• 4. Agree matching and survivorship rules with the data owners. Decide which evidence makes two records the same customer, which record survives, and which value wins for each column, and get the business owners of the data to sign it off before anything merges.
• 5. Tune duplicate detection rules to the real data. Build the published rules from the evidence that profiling showed is reliable, ignore blank values, run duplicate detection jobs to confirm the rules find the groups profiling found, and switch detection on for create, update and import.
• 6. Snapshot reports and related records before each batch. Save the key report figures and, for every record in the batch, the IDs of its related activities, cases, opportunities, orders and invoices, so the batch can be reconciled and reversed.
• 7. Merge in small, logged batches. Merge the strongest matches first, record the survivor, the subordinate and the moved child records for every merge, and hold weaker matches for review by the people who know the customers.
• 8. Reconcile reports after every batch. Compare the saved figures with the new ones and explain every difference before the next batch, including reports and exports that key on the subordinate record ID.
• 9. Add alternate keys once the key column is clean. Create a normalised identifier column, such as registration number or normalised email, back-fill it, and define an alternate key so the platform rejects a second record with the same value.
• 10. Put intake controls on every route and monitor. Make integrations upsert on the alternate key, keep detection on for imports and app entry, configure matching on marketing forms, and schedule a recurring duplicate detection job with a named owner for the results.

Who does this work, and is Dynamics 365 the right place for this data?

Solzet delivers duplicate cleanup as a scoped Dynamics 365 Customer Engagement and Power Platform engagement, remotely from Yerevan, Armenia: the route inventory, profiling, rules, the logged merge batches, alternate keys and intake controls, with your data owners making the matching decisions. Where duplicates are one symptom of a broken build, the project rescue and takeover service is the better starting point.

We recommend the right solution - whether that's Microsoft Dynamics 365, Power Platform, or a custom-built CRM. Some businesses need the Microsoft ecosystem. Others need full control without licensing. We deliver both. If per-user licensing or hosting requirements are part of why data is scattered across spreadsheets and other tools, a custom CRM on React, Node.js, PostgreSQL or .NET can enforce unique customer identity in its own database constraints from the first day.