Fast Dataverse Bulk Import: Strategies for Millions of Rows
A technical guide for anyone watching a two million row load crawl into its third day: what the service protection limits really are, what the arithmetic on them says, and the order in which to fix things.
The fastest way to load two million rows into Dataverse is tiered, and the three day wait is usually not the API limits. Cut the rows you do not need, stage and clean outside the platform, then disable everything that fires per row: plugins, flows, workflows, business rules, duplicate detection, auditing. Write with the bulk messages, CreateMultiple and UpsertMultiple, not one request per row. Parallelise to what the platform tells you and treat throttling as flow control, not failure. Use KingswaySoft or Azure Data Factory where the mapping is complex. This is how we actually run large Dataverse migrations.
Two notes before the detail. First, we do not publish throughput benchmarks. Rows per second on a Dataverse load depends on how wide the table is, how many lookups each row resolves, how much logic is still switched on, and what else is running in the environment, so any number quoted without seeing your environment is decoration. What we can give you is the arithmetic on the published limits, which is what actually tells you whether three days is inevitable or self inflicted. Second, the service protection limits, bypass headers, and bulk message names below are Microsoft platform behaviour, and Microsoft revises all three. Check the current Dataverse developer documentation for the exact figures and header names before you build a plan around them.
The answer in three lines
The single biggest win is the request count
One create request per row is what turns two million rows into a multi day job. The bulk messages, CreateMultiple, UpdateMultiple, and UpsertMultiple, carry many rows in one request, and that one change moves the load from millions of requests to tens of thousands. Nothing else on this page comes close to it.
The second biggest win is turning things off
Every synchronous plugin, real time workflow, business rule, duplicate detection rule, rollup, and audit write fires again on every row. A load that takes three days is very often a load that is quietly running your entire business logic two million times. Disabling that for the duration of the run is not a hack, it is the standard migration procedure.
Throttling is a speed limit, not a wall
Service protection limits return HTTP 429 with a Retry-After header. A load that backs off and retries on that signal keeps running at the fastest rate the platform will accept. A load that treats 429 as a row failure produces a broken dataset and a night of manual repair. The limits are not the reason you are slow, but mishandling them is.
Why it takes three days, and why it is usually not the API limits
When we are handed a load that will not finish, the cause is almost never the throttling message in the log. The throttling is what the loader noticed. These are the things that were actually costing the time.
One request per row
The default shape of most hand written loaders, and of a lot of tooling used without tuning. At two million rows that is two million round trips, each one paying network latency, authentication overhead, and the platform pipeline. This is the first thing to check and it is usually the answer.
Business logic firing two million times
Synchronous plugins, real time workflows, business rules, calculated and rollup columns, duplicate detection, and auditing all run per row. None of them were written with a bulk load in mind. Worse, some of them will overwrite the values you are trying to load, so leaving them on costs you both speed and correctness.
Lookups resolved by name
Every relationship resolved by searching for a matching account name or contact email is an extra query per row, and often more than one. Resolving relationships through an alternate key on the source system identifier turns that into part of the write itself. This alone can be the difference between hours and days.
A single thread doing all the work
Dataverse is designed to be written to concurrently. A serial loader leaves most of the available capacity idle while it waits for each response, and the limits it eventually complains about were never the constraint. The fix is controlled parallelism, not a bigger batch on one thread.
Retry logic that gives up or hammers
Two failure modes, both expensive. A loader that stops on the first 429 turns a throughput signal into an outage. A loader that immediately retries without honouring Retry-After extends its own throttling and slows everything else in the environment at the same time.
Files and attachments counted as rows
File and image content is not the same workload as relational rows. It is larger, it encodes badly, it consumes file capacity rather than database capacity, and it is almost always the slowest part of a migration. Planned inside the same run as the row load, it is what makes the estimate wrong.
Loading rows nobody will ever read
The fastest two million row import is the one where four hundred thousand rows turned out to be enough. Closed activities from nine years ago, records from a product line that no longer exists, and audit style tables that were never queried are all volume you can archive elsewhere instead of paying for on every rehearsal.
The arithmetic on the published limits
Everything in this section is Microsoft's documented service protection limits plus division. These are ceilings, not predictions: they tell you what is impossible rather than what is achievable, which is exactly the question when somebody asks whether three days is normal.
What the limits actually are
Dataverse service protection limits are applied per user, per web server node, across a five minute sliding window. The published thresholds are six thousand requests, twenty minutes of combined execution time, and fifty two concurrent requests. Exceed one and the platform returns HTTP 429 with a Retry-After header telling you how long to wait. These are separate from the licensed API request entitlements, which are a different mechanism with different consequences.
Why one request per row cannot finish quickly
Six thousand requests every five minutes is seventy two thousand an hour for one user on one node. Two million rows at one request per row is two million requests, which is roughly twenty eight hours of pure limit bound time even if every request were instant and nothing else ever went wrong. Add per row business logic and a single thread and three days is not surprising, it is arithmetic.
Why the bulk messages change the shape of the problem
Carry a few hundred rows in each CreateMultiple call and two million rows becomes a five figure number of requests rather than a seven figure one. The request count stops being the binding constraint almost immediately, which is why this is the first change we make on any load that is running long.
The execution time budget is the one people miss
Twenty minutes of combined execution time inside a five minute window is the equivalent of about four requests executing continuously at once, per user, per node. That is the number that explains why adding twenty threads does not make a load five times faster. Beyond that point you are not gaining throughput, you are generating 429s and spending the difference on backoff.
Per user and per node, which cuts both ways
Because the window is per user, spreading a migration across several dedicated application users genuinely raises the ceiling, and it is a normal pattern on large loads. Because it is also per web server node and you cannot see or choose the nodes, it is not a number you can plan against precisely. Use it as headroom, never as the reason a badly shaped load is expected to finish.
Let the platform tell you the thread count
Dataverse returns a recommended degree of parallelism in a response header, and the modern SDK client surfaces it as a property. Reading that and sizing your worker pool from it is more reliable than guessing, because it reflects the capacity actually available to you at that moment rather than the capacity you hoped for when you wrote the config file.
The tiered strategy, step by step
Steps 1 to 4 decide how much work there is. Steps 5 to 10 do the load. Steps 11 and 12 prove it and hand it back. The order matters, because tuning batch sizes on a load that is still firing plugins is optimising the wrong number.
Decide how many rows you are really importing
Profile the source table by table and by age before anyone opens a mapping tool. Agree in writing what is migrated live, what is archived to a data lake, a SQL database, or a read only copy of the old system, and what is deliberately dropped. Every row removed here is removed from every rehearsal as well as from the cutover, and this is the only step that makes the load faster without making it more complicated.
Stage and clean the data outside Dataverse
Do the joins, deduplication, type conversion, option set mapping, and lookup resolution in a staging database or a data lake, where set based operations are cheap. Dataverse is the slowest place to do transformation work and the most expensive place to correct it. What arrives at the platform should be rows that are already correct and already know the identifier of every record they relate to.
Put an alternate key on the source identifier
Add a column for the source system identifier to every migrated table and define an alternate key on it. That single decision gives you three things: relationships resolved by key instead of by a lookup query per row, upsert semantics so a re-run repairs rather than duplicates, and a restartable load. Keep the keys you need for the migration under review afterwards, because every alternate key is an index the platform maintains on every future write.
Turn off everything that fires per row
Disable synchronous plugin steps, real time workflows, Power Automate flows, business rules, duplicate detection rules, and SLAs for the duration of the load, and turn auditing off so the migration does not spend capacity recording itself. Where the logic cannot be disabled globally, Dataverse also supports request level bypass options for custom plugin and flow execution, which require an elevated privilege granted to the migration account. Write down everything disabled, because re-enabling it has to be a checklist rather than a memory test.
Load in dependency order
Users, business units, and teams first, then currencies and reference data, then parents before children, then activities, then files. Resolve every relationship through the alternate key rather than a name or an email address. Loading out of order is what produces orphaned lookups discovered a week later, and repairing those costs more time than the whole load saved.
Write with the bulk messages, not one row at a time
Use CreateMultiple, UpdateMultiple, and UpsertMultiple so each request carries many rows. This is the change that moves a two million row load from millions of requests to tens of thousands. ExecuteMultiple is a different mechanism: it reduces round trips by grouping requests, but the server still processes them one at a time, so treat it as a convenience rather than as the throughput answer.
Size the payload by measurement, not by hope
There is no universally correct batch size, because it depends on how wide the rows are and how much work each one causes. Start modest, measure, and increase until throughput stops improving. Oversized payloads do not fail gracefully, they fail as request timeouts after doing most of the work, which is the worst possible outcome on a load that has to be restartable.
Parallelise to what the platform recommends
Run several workers writing concurrently and size the pool from the recommended degree of parallelism the platform returns, rather than from a number in a configuration file. Remember the execution time budget: past a certain concurrency you are producing throttling responses instead of throughput. If you genuinely need more headroom, spread the load across several dedicated application users rather than pushing one user harder.
Treat throttling as flow control
On HTTP 429, wait for the interval in the Retry-After header and retry the same payload. Never count a 429 as a row failure, never retry immediately, and never let a throttled worker take the whole run down. A load that handles throttling correctly runs at the fastest rate the platform will accept, which is the actual definition of fast here. A load that does not cannot survive a large table.
Run files and attachments as their own workstream
Notes, attachments, and file and image columns are a separate object graph with separate capacity and different throughput characteristics, and they are usually the longest running part of the migration. Decide where they land, whether that is Dataverse file columns, SharePoint, or Azure Blob Storage with the reference held in Dataverse, batch them with per file logging, and reconcile file counts and total bytes per parent table.
Rehearse at full volume and measure
Run the complete sequence end to end against full production volume in a sandbox, timing each table. That rehearsal is what converts an estimate into a duration you can plan a cutover weekend around, and it is where you find the table whose plugin was never disabled. Bear in mind a sandbox is not always sized identically to production, so treat the rehearsal as a shape and a bottleneck map as well as a clock.
Reconcile, then re-enable in a written order
Row counts per table, control totals on the numeric columns that matter, and a referential integrity check that no lookup is empty where the source had a value. Only once that passes do you re-enable plugins, flows, workflows, duplicate detection, SLAs, and auditing, in a deliberate order with a check after each. Turning auditing back on after the load rather than before means the audit log starts clean on day one.
Choosing the tool
On a large migration the honest answer is usually a combination rather than one product. Each of these earns its place somewhere, and each of them costs you something.
| Tool | What it is good at | What it costs you |
|---|---|---|
| KingswaySoft SSIS Integration Toolkit | The default for complex Dynamics 365 Customer Engagement maps. Dataverse aware components handle the platform specific parts natively: upsert on alternate keys, lookup resolution, impersonation, notes and attachments, per row error output, and multi threaded writes with batch sizes you control from the component rather than from code. | Licensed, and it needs SSIS and Visual Studio, so somebody has to own that environment. It is a developer tool rather than something a business analyst drives, and the quality of the package design decides your throughput just as much as the product does. |
| Azure Data Factory or Synapse pipelines | Suits a staged, cloud native shape: large extracts landed in storage or SQL, transformed at scale, then written through the Dataverse connector with configurable write batch size and concurrency. It doubles as the integration platform after go live, which is often the deciding argument. | The transformation work is where the effort goes, and the Dataverse specific mechanics that KingswaySoft ships as components are things you assemble yourself. Cost is consumption based, so a badly shaped pipeline is visible on the bill as well as on the clock. |
| Dataverse dataflows (Power Query) | Low ceremony, no developer environment, and a familiar transformation experience. The right tool for reference data, option set backing tables, and corrective loads, and a reasonable answer for a one off table in the tens of thousands of rows. | Not the tool for a multi million row load with a complex relational map. Throughput and control over batching and parallelism are limited compared with the alternatives, and debugging a long running refresh is harder than reading a per row error log. |
| Custom code against the Web API or the SDK | Complete control, which is exactly what the awkward parts need: the bulk messages with your own payload sizing, the recommended parallelism header honoured properly, request level bypass options, impersonated creates, and the state and closure pass that no tool does well. | You own everything, including retry on service protection limits, batching, per row logging, and restartability. Written casually it becomes the slowest and least observable part of the project, and the one nobody else can run at three in the morning. |
| The classic data import API and import wizard | Server side and asynchronous, with source file mapping, duplicate detection, and a per row failure log surfaced in the application. Genuinely useful for reference data and for imports a functional consultant needs to run without a developer. | It processes records through the standard pipeline, so business logic still fires per row unless you have disabled it, and it is not built to be the fastest path for millions of rows. Treat it as an operational tool rather than as the migration engine. |
| Elastic tables | Built for very high volume ingest and horizontal scale, with a partition key you choose and automatic scaling. The right answer for telemetry shaped, append heavy, high cardinality data where the write rate is the whole problem. | A different storage model with real trade offs: relationship, rollup, and transaction behaviour differ from standard tables, and query patterns have to suit the partition key. It is a design decision for a specific class of data, not a switch you flip to make a customer table load faster. |
If the source is Salesforce specifically, the mapping and loss decisions matter more than the throughput, and we work through those in our Salesforce to Dynamics 365 data migration guide.
What to switch off for the load
This is the checklist that recovers the most time, and it is a correctness measure as much as a speed one. Logic written for a user typing into a form does not behave sensibly when two million rows arrive at once.
Synchronous plugin steps on every table in scope, and the asynchronous ones too where they exist only to enrich a record you are already loading correctly.
Real time and background workflows, including the ones nobody remembers registering, which is why this is an inventory exercise rather than a memory exercise.
Power Automate flows with a Dataverse trigger on the tables being loaded. Two million rows arriving is two million trigger evaluations, and the flow run history alone becomes a problem.
Business rules, which will happily overwrite values you are loading and do it silently.
Duplicate detection rules, which add a matching query to every write and will reject legitimate historical rows.
Auditing, which roughly doubles the write work and consumes log capacity recording a migration nobody will ever audit. Turn it on after the load so day one starts clean.
SLAs and any rollup or calculated column refresh you can defer, so the platform is not recalculating aggregates while you are still inserting the rows they aggregate.
And how to switch it back on
The half of the checklist that gets skipped, and the one that produces the incident on the Monday after a successful cutover.
Re-enable in the reverse order and check after each one, rather than switching everything back and hoping. The check is cheap and the alternative is discovering the failure through a user.
Auditing first, before anyone touches the data, so the log has a clean starting point and the first real user change is the first audited change.
Then business rules and duplicate detection, then workflows, then plugins, then flows, so the noisiest components come back last and against data that has already been validated.
Run a small deliberate transaction end to end afterwards: create a record, edit it, and watch the automation behave, because the migration account and a real user do not always see the same platform.
Reconcile once more after re-enabling. Automation coming back on can change data, and it is better to know that on the Saturday than on the Monday.
How Solzet runs a load of this size
Data migration is part of our Dynamics 365 Customer Engagement and Power Platform delivery rather than a product we sell separately, so this is what the work looks like when we do it.
We profile before we promise a duration
Row counts, table widths, relationship depth, file volume, and the automation inventory on every table in scope. That profile is what tells us whether your load is limit bound, logic bound, or simply carrying rows nobody needs, and those three problems have three different fixes.
We build the load to be restartable from the first line of code
Alternate keys on source identifiers, upsert rather than create, per row logging, and the ability to resume from a failed batch. The value of a restartable load only becomes obvious the first time a batch fails at three in the morning, which is exactly when it is too late to add it.
We treat throttling as part of the design, not as an incident
Backoff on Retry-After, worker pools sized from the parallelism the platform recommends, payload sizes tuned by measurement, and dedicated application users where the headroom is genuinely needed. The target is the fastest rate the platform will accept, sustained, rather than a burst followed by an hour of 429s.
We use the right tool per workstream rather than one tool for everything
On a large migration the honest answer is usually a combination: a purpose built toolkit or a pipeline platform for the bulk relational load, and custom code against the Web API for files, impersonated creates, and the state and closure pass. Insisting on a single tool is how the awkward twenty percent ends up costing eighty percent of the time.
We rehearse at full volume before anyone books a cutover weekend
The complete sequence end to end in a sandbox at production volume, timed per table, with the reconciliation run as part of the rehearsal rather than after it. That is what turns a plan into a schedule the business can actually make decisions around.
We build on proper solution lifecycle management
Separate development, test, and production environments, changes made in unmanaged solutions and promoted as managed ones, so what was disabled for a load is a versioned, reviewable change rather than an edit somebody made live and forgot.
We stay inside our scope and say where it ends
We deliver Dynamics 365 Customer Engagement and the Power Platform: Sales, Customer Service, Field Service, and Customer Insights on Dataverse, plus Power Apps, Power Automate, and PCF. We do not implement Microsoft finance and operations products, so where a migration reaches the ledger we scope the integration to whichever finance system you run rather than claiming the whole stack.
Certified engineers in one time zone
We deliver from a single hub in Yerevan, Armenia, at GMT+4, a working day that overlaps Western European hours and reaches into the US morning, which matters more than usual when a cutover runs through a weekend. Our engineers hold Microsoft certifications including PL-200, PL-400, and PL-600 for the Power Platform.
More detail is on our Dynamics 365 consulting page, and if the load you are fighting was inherited from a project that has already stalled, that is a different conversation, covered under rescue and project takeover.
Frequently Asked Questions
What is the fastest way to bulk import two million rows into Dataverse without hitting API limits?
Work through it in tiers, in this order. Cut the row count first, because rows you archive elsewhere cost nothing to load. Stage and clean the data outside Dataverse so what arrives is already correct and already knows the key of every record it relates to. Disable everything that fires per row: plugins, real time workflows, Power Automate flows, business rules, duplicate detection, and auditing. Write with the bulk messages, CreateMultiple and UpsertMultiple, so each request carries many rows rather than one. Then parallelise to the degree of parallelism the platform recommends, and back off properly on HTTP 429 using the Retry-After header. Done in that order, service protection limits usually stop being the binding constraint before you get to the end of the list.
Why does our Dataverse import take three days when the data is not that large?
Almost always one of five things, and rarely the API limits themselves. One request per row, so two million rows means two million round trips. Business logic firing on every row, so your entire plugin and flow estate runs two million times. Lookups resolved by searching for a name instead of by an alternate key, which adds a query per row. A single thread doing all the work while most of the available capacity sits idle. Or retry logic that treats a throttling response as a failure and stalls. The fastest way to find out which one applies to you is to time a single table with logging on and look at where the time actually goes, rather than assuming it is the platform.
What are the Dataverse service protection limits exactly?
They are applied per user, per web server node, across a five minute sliding window, and the published thresholds are six thousand requests, twenty minutes of combined execution time, and fifty two concurrent requests. Cross one and Dataverse returns HTTP 429 with a Retry-After header telling you how long to wait. These are separate from the licensed API request entitlements that come with your licences, which are a different mechanism. The practical reading of the execution time threshold is the useful part: twenty minutes of execution inside a five minute window is roughly four requests running continuously at once for one user on one node, which is why adding many more threads produces throttling rather than throughput. Microsoft revises these figures, so confirm them in the current documentation before you plan around them.
Should we use CreateMultiple or ExecuteMultiple for a bulk load?
CreateMultiple, and UpsertMultiple where you want a re-runnable load. They are the messages built for writing many rows in one request, and moving to them is the single largest throughput change available on most loads. ExecuteMultiple solves a different problem: it groups arbitrary requests to save round trips, but the server still processes them one after another, so it is a convenience rather than a throughput mechanism. If a load is currently issuing one create per row, switching to the multiple messages will do more for it than any amount of thread tuning.
Do we have to disable plugins and flows to import data quickly?
For a load of this size, yes, and it is standard migration procedure rather than a shortcut. Synchronous plugins, real time workflows, business rules, duplicate detection, and Power Automate flows all execute per row, so on two million rows they are the dominant cost. They are also a correctness problem, because logic written for a user typing into a form will happily overwrite created dates, ownership, and status values you are trying to preserve. Where a component cannot be disabled globally, Dataverse supports request level bypass options for custom plugin and flow execution that require an elevated privilege on the migration account. Whatever route you take, write down what was switched off, and re-enable it in a deliberate order with a check after each one.
How do we handle throttling and 429 responses during a migration?
Treat them as flow control. When Dataverse returns HTTP 429, read the Retry-After header, wait for that interval, and retry the same payload. Never count a throttled request as a row failure, because that produces a dataset with holes in it and a night of manual repair. Never retry immediately either, because that extends your own throttling and degrades the environment for everyone else in it. A well built loader is throttled routinely and nobody notices: it is simply running at the fastest rate the platform will accept. If you are seeing throttling constantly rather than occasionally, that is a signal to reduce concurrency or increase rows per request, not a signal to retry harder.
Is KingswaySoft worth it, or should we write our own loader?
On a complex Dynamics 365 Customer Engagement map, KingswaySoft inside SSIS is the default for good reasons: the Dataverse aware components handle upsert on alternate keys, lookup resolution, impersonation, notes and attachments, per row error output, and multi threaded writes, and every one of those is a thing you would otherwise build and debug yourself. The trade off is the licence and the fact that somebody has to own an SSIS and Visual Studio environment. Azure Data Factory is the better fit when the shape is already cloud native and you want the same platform for integration after go live. Custom code against the Web API usually still ends up owning the awkward twenty percent regardless of which main tool you choose, which is files, impersonated creates, and the state and closure pass. On a large migration the honest answer is normally a combination rather than a single tool.
Can we use elastic tables to make a large import faster?
For the right kind of data, yes, and for a normal business table, no. Elastic tables are built for very high volume, append heavy, high cardinality workloads such as telemetry, with a partition key you choose and automatic scaling behind it. If that describes the data you are loading, they are a genuine answer to a write rate problem. If you are migrating accounts, contacts, opportunities, and their history, they are not, because relationship, rollup, and transaction behaviour differs from standard tables and your query patterns have to suit the partition key. It is a data design decision made for a specific class of data, not a performance switch for an existing model.
How long should a two million row Dataverse migration actually take?
It depends on how wide the tables are, how many relationships each row resolves, how much file content comes with it, and how much logic can be disabled, so any duration quoted without seeing the environment is guesswork and we will not put one on a page. What is predictable is where the time goes: profiling and the mapping take longer than people expect, the bulk relational load is usually the fastest part once it is shaped correctly, files are usually the slowest, and the reconciliation is what decides whether you can sign the cutover off. The way to get a real number is a full volume rehearsal in a sandbox, timed per table. That converts an estimate into a measured duration you can plan the business around, and it is the deliverable we would push for before anyone books a weekend.
Send us the load that will not finish
Tell us the row counts, the tables, what is still switched on, and how the loader is writing. You will get a straight assessment of whether the problem is the limits, the logic, or the volume, and what the realistic duration looks like once it is shaped properly. Solzet is a Dynamics 365 Customer Engagement and Power Platform consultancy in Yerevan, Armenia, working with clients and Microsoft partners across Europe and the US.