Diagnosing Intermittent Dynamics 365 Failures Nobody Can Reproduce

A troubleshooting reference for plug-in races, two-minute timeouts, null references and ordering faults that only appear under load: gather the evidence, reproduce it on purpose, contain the damage and fix the design.

Intermittent Dynamics 365 failures persist because teams guess. Start with evidence: capture the correlation ID and timestamp from a failing request, pull the plug-in trace log rows and the System Jobs record for that same ID, and line them up against what else was running at that moment. The usual causes under load are plug-ins and real-time workflows racing on the same record, execution order assumptions that only hold when everything is fast, null references on records another asynchronous process has not yet created, the two-minute plug-in sandbox timeout on bulk work, and async queue backlog. Then reproduce deliberately with concurrent load in a sandbox, and instrument so the next occurrence is diagnosable rather than anecdotal.

Why do Dynamics 365 errors disappear when you try to reproduce them?

Because the person reproducing them is one user, saving one record, on a quiet sandbox. The failures that matter in production depend on things a single manual test never recreates: two processes touching the same record within the same second, an integration sending a partial update rather than the full form, an asynchronous job running minutes later than usual because the queue is busy, or a bulk update pushing a plug-in loop past its time limit.

That is why these problems survive for months. Each report arrives as "it failed for someone this morning", someone tries it, it works, and the ticket is closed as user error. The general design mistakes behind many of these faults, such as slow synchronous plug-ins and logic scattered across flows and plug-ins, are catalogued on our Dynamics 365 customisation mistakes page. This page is the diagnostic method for when one of them is already hurting you.

What evidence should you capture before anyone forms a theory?

Evidence discipline is the whole difference between a fix and another month of guessing. Agree one rule with the business: every failure report must come with the time, the record, the user and the error details, and nobody changes configuration until that evidence is pulled. Then work through the sources below for the same operation.

SourceWhat to pullWhat it tells you
The error itselfThe exact message, the timestamp with time zone, the record ID, the user, and the downloadable log file from the error dialog, which carries the session and activity identifiers.Which operation failed and when, so every other source can be filtered to that moment.
Plug-in trace logRows for the same time window and record, including correlation ID, request ID, depth, mode, message name, primary entity, execution start and duration, and exception details.Which plug-ins ran for that operation, in what order, how deep in the call chain, and how long each took.
System JobsAsynchronous plug-in and workflow jobs for the record: status, status reason, created, started and completed times, error message and correlation ID.Whether async work failed, waited in the queue, or ran after something that depended on it.
Audit historyChanges to the record and related records around the failure, if auditing is enabled for those tables and columns.Which process or user wrote which value, and in what order, which is how races show up.
Flow run historyRuns of any Power Automate cloud flow triggered by the same table in that window.Whether a flow updated the record between a plug-in reading it and writing it.
Integration logsRequests from middleware or external systems to the same record, with their payloads.Whether a partial update or a burst of concurrent requests arrived at the same time.

How do you use the correlation ID to follow one failing operation?

Every plug-in execution context carries a correlation ID, and it is shared by the operations that one original request sets off, including the plug-ins it triggers and the asynchronous jobs queued from it. That makes it the thread you pull. Find the trace log row for the exception, copy its correlation ID, then filter the trace log and System Jobs by that ID. What comes back is the chain of work caused by one save, rather than everything that happened in that minute.

  • Sort the trace rows by execution start time and read the depth column. Depth 1 is the original request; higher depths are operations caused by other plug-ins or workflows, which is where loops and unexpected cascades appear.
  • Compare the mode column: synchronous rows ran inside the save, asynchronous rows ran later from the queue, possibly after other work changed the record.
  • Check the duration column for any plug-in close to two minutes. That is the sandbox limit, and a step that takes most of it on a quiet day will exceed it under load.
  • Look for two correlation IDs touching the same record within seconds. Two independent operations on one record is the signature of a race.
  • Keep the IDs in the ticket. Once a fix ships, the same query shows whether the pattern has stopped.

Is the plug-in trace log switched on, and how long does it keep evidence?

Often not, which is why the first occurrence is lost. The setting lives in the system settings of the environment as "Enable logging to plug-in trace log", with three values: Off, Exception and All. Exception writes a row only when a plug-in throws, which is the sensible production default. All writes a row for every execution and is useful for a short, planned diagnostic window, but it adds load and volume and should be switched back afterwards.

Trace log rows are cleared automatically by a system bulk deletion job, by default after about a day, so they are not an archive. When a significant failure happens, export the relevant rows the same day and attach them to the incident. A trace row is also only as useful as what the plug-in wrote into it: plug-ins that call the tracing service with the record ID, the branch taken and the values read make the next occurrence readable, while plug-ins that only throw a generic message leave you with a stack trace and nothing else.

What causes Dynamics 365 plug-ins to fail only under load?

Almost every intermittent plug-in fault falls into one of a small number of patterns. Each has a recognisable trace, so match the evidence to the pattern rather than starting from the code.

CauseWhat the evidence looks likeWhy it only happens under load
Race between a plug-in and a real-time workflow or flow on the same recordTwo writers to the same columns within seconds, visible in audit history; the final value depends on which finished last.When the system is quiet one always finishes first. When it is busy the order changes.
Execution order assumptionTwo steps on the same message and stage with the same execution order, where one reads a value the other sets.The platform does not guarantee an order between steps with equal execution order, so the working order is luck.
Null reference on a record created asynchronouslyA NullReferenceException or "does not exist" error on a retrieve of a related record or lookup that is set by an async plug-in, workflow or flow.The async job normally runs before anyone opens or updates the record. With a queue backlog it runs later.
Partial update payloadThe Target on Update is missing a column the plug-in reads directly, and there is no pre-image registered for it.Forms send the changed columns; integrations and bulk updates often send only one or two columns.
Shared state inside the plug-in classWrong values or errors that match another record being processed at the same moment.Plug-in instances are cached and can serve concurrent requests, so class-level fields leak data between executions.
Two-minute sandbox timeoutThe error says the plug-in execution timed out in the sandbox host; the duration is at or near two minutes.A loop over child records, or an external call, grows with data volume and contention.
Recursion and cascadesIncreasing depth values in the trace log for one correlation ID, sometimes ending in a loop detection error.Each update triggers another plug-in or workflow that updates the record again, and bulk work multiplies it.
Async queue backlogSystem Jobs sitting in Waiting or Waiting For Resources long after creation.Bulk operations and chatty async steps fill the queue, delaying everything else in the organisation.

Why do plug-ins fire in the wrong order and corrupt data?

Usually because nobody decided the order. Each registered step has a stage, a mode and an execution order, and together they are the only contract you have. Pre-validation runs before the main system operation and, for the original request, outside the database transaction. Pre-operation runs inside the transaction before the record is written, which is where you change values on the Target. Post-operation runs after the write; asynchronous steps can only be registered here, and they run later from the queue in their own transaction.

Within one message and stage, steps run in ascending execution order. When two steps share the same value, their relative order is not guaranteed, and real-time workflows on the same message add more writers whose timing relative to your plug-ins you should not rely on. The corruption appears when one step reads a value another step was supposed to set first, or when an asynchronous step overwrites a value a later synchronous save already corrected.

  • List every step, real-time workflow and automated cloud flow on the affected table and message, with stage, mode, execution order and filtering attributes, in one table.
  • Give every step an explicit, distinct execution order and write down why it runs where it does.
  • Set values in pre-operation on the Target instead of issuing a separate update in post-operation, which triggers the pipeline again.
  • Keep one owner per column: if a plug-in and a flow both set the same column, decide which one owns it and remove the other.
  • Narrow filtering attributes so update steps only fire when the columns they care about change.

How do you handle the two-minute plug-in timeout on bulk updates?

Plug-ins run in the sandbox with a two-minute execution limit, and it applies whether the step is synchronous or asynchronous. You cannot raise it. A plug-in that updates every related record in a loop, or waits on an external service, will pass every test with ten children and fail on the account with thousands, or when a bulk update from an integration arrives while the system is busy.

The fix is to stop doing unbounded work inside one plug-in execution. Keep the synchronous step to validation and the values that must be correct at save time, and hand the rest to a process built for volume.

  • Move fan-out work, such as updating every child record, out of the plug-in and into a queued process: an Azure Service Bus queue fed by a service endpoint and processed by Azure Functions, or a Power Automate flow that pages through records in chunks.
  • Process in bounded batches, each well inside the limit, and record progress so a failed batch can resume rather than start again.
  • For integrations and data fixes that send many requests, group them with ExecuteMultiple or the CreateMultiple and UpdateMultiple messages, and respect service protection limits by backing off when the service asks you to. Throughput for large loads is covered in our Dataverse bulk import guide.
  • Remember that each record in a bulk operation still runs its plug-ins, so a slow step is multiplied by every row. Make the step fast before making the batch bigger.
  • Never call slow external services synchronously from a plug-in. Queue the call and write the result back.

How do you reproduce an intermittent failure deliberately?

Recreate the conditions, not the click. Use a sandbox copied from production so the data volumes, related record counts and registered steps match, and switch the trace log to All for the duration of the test only.

  • Replay the real payload: send the same partial update the integration sends, not a form save with every column.
  • Add concurrency: run the same operation against the same record from several parallel clients at once, using a small load harness or test tool that calls the Web API.
  • Add delay: pause or slow the asynchronous job the failing step depends on, so the dependent record does not exist yet when the synchronous step runs.
  • Add volume: run the bulk update against a record with the realistic maximum number of children, and watch plug-in durations approach the two-minute limit.
  • Keep a written script of each run with its correlation IDs, so a reproduction becomes a regression test once the fix is in.

What should you do first if data is being corrupted right now?

Contain before you fix, and do not destroy the evidence you need for both the fix and the data repair. Work in this order, logging each change with its time.

StepHowWhy
1. Export the evidenceExport the trace log rows, failed System Jobs and audit history for the affected records and time window.Trace rows are cleared automatically and the next steps change behaviour.
2. Deactivate the suspect stepDisable the step in the Plugin Registration Tool, or deactivate the real-time workflow or flow. Do not unregister the assembly or delete the step.Registration, filtering attributes and images stay in place for a controlled re-enable.
3. Pause dependent async workSuspend or cancel queued System Jobs that would keep writing bad values, and pause integrations writing to the table if needed.Stops the damage spreading while the cause is confirmed.
4. Tell users what changedExplain which automatic behaviour is off and what they should do by hand meanwhile.Avoids people re-entering data that the paused process would duplicate later.
5. Scope the damageUse audit history and the correlation IDs to list every record written by the faulty chain.Gives the repair a defined list instead of a guess.
6. Repair, then re-enableCorrect records from audit history in a controlled script, ship the fix through a managed solution, re-enable, and watch the trace log.Repairing before the fix ships means repairing twice.

How do you fix these faults so they stay fixed?

Most permanent fixes are small and structural rather than clever. The aim is that the design no longer depends on timing.

  • Explicit stage and execution order on every step, documented next to the code, with no two steps on one message and stage sharing a value.
  • Guarded retrieves: check that a related record or lookup exists before using it, and when it legitimately may not exist yet, defer the work to an asynchronous step or a retry rather than throwing.
  • Pre-images and post-images registered for every column a step reads, so partial updates do not produce nulls.
  • Stateless plug-in classes: no instance or static fields holding per-execution data.
  • Depth checks where a step can trigger itself, and filtering attributes on every update step.
  • Consolidated overlapping logic: one plug-in or one flow owning a column, instead of a plug-in, a real-time workflow and a flow each doing part of it.
  • Asynchronous chunking for any work that grows with data volume.
  • Tracing in every plug-in that writes the record ID and the decision taken, with the trace log left on Exception in production, and telemetry sent to Application Insights where your environment supports it.

When is the problem the platform choice rather than the plug-ins?

Rarely, for this class of fault. Races, timeouts and ordering faults come from design, and they appear on any platform with triggers, background jobs and concurrent writers. Moving to another system without fixing the design moves the problem with it. If the investigation shows the environment has much wider issues, an independent Dynamics 365 health check and technical audit gives you the full picture and the evidence for it before any decision to rebuild.

We recommend the right solution - whether that's Microsoft Dynamics 365, Power Platform, or a custom-built CRM. Some businesses need the Microsoft ecosystem. Others need full control without licensing. We deliver both. Where the evidence points to a process Dynamics 365 licensing and its customisation model do not suit, a custom-built CRM on React, Node.js, PostgreSQL or .NET is the alternative, designed with its own queues and transactions from the start.

How does Solzet diagnose and fix intermittent Dynamics 365 failures?

We work in the order of this page. First the evidence: trace logging, correlation IDs, System Jobs and audit history for real failures, pulled before anyone changes a thing. If data is being damaged, containment alongside your team, with each change logged and reversible. Then a map of every step, workflow and flow on the affected tables, a deliberate reproduction under concurrent load in a sandbox, the fix shipped through managed solutions, and the data repair.

We leave behind tracing in the code, a documented step register and the reproduction scripts as regression tests. Solzet delivers remotely from Yerevan, Armenia, with senior consultants and full-stack developers and 8+ years of Dynamics 365 Customer Engagement and Power Platform work, directly or white-label for Microsoft partners. The plug-in and Dataverse development side is described on our Dynamics 365 development service.

What do people ask us?

How do you diagnose a Dynamics 365 error nobody can reproduce?

Start from evidence, not theories. Capture the timestamp, record, user and error details of a real failure, then pull the plug-in trace log rows and System Jobs for the same correlation ID and time window, plus audit history for the record. Line them up to see which steps ran, in what order and how long they took. Then reproduce in a production-like sandbox with the real payload, concurrency and volume.

What is the plug-in execution timeout in Dynamics 365?

Plug-ins run in the sandbox with a two-minute execution limit, for both synchronous and asynchronous steps, and it cannot be raised. When a plug-in exceeds it, the operation fails with a timeout error from the sandbox host. The fix is to keep plug-ins short and move work that grows with data volume, such as updating many child records or calling external services, into queued, chunked processing.

Why do we get a null reference in a Dynamics 365 plug-in only under load?

Usually because the plug-in reads something that is not always there. Common causes are a related record created by an asynchronous job that has not run yet because the queue is busy, a partial update where the Target lacks a column and no pre-image is registered, or class-level fields shared between concurrent executions. Guard retrieves, register images, and keep plug-in classes stateless.

How is the execution order of Dynamics 365 plug-ins decided?

By stage first, then by the execution order value on each step. Pre-validation runs before pre-operation, which runs before the write, then post-operation. Within one message and stage, steps run in ascending execution order, and steps with the same value have no guaranteed order. Asynchronous steps run later from the queue. Give every step a distinct execution order and avoid mixing real-time workflows with plug-ins on the same columns.

Should the plug-in trace log be set to All in production?

Not permanently. Exception is the sensible production setting: it records a row whenever a plug-in throws, without the load and volume of logging every execution. Switch to All for a short, planned diagnostic window when you need to see successful executions and their timings, then switch back. Trace rows are cleared automatically, so export anything important the same day.

What should we do first when a plug-in is corrupting data?

Export the trace log rows, failed System Jobs and audit history for the affected window, then deactivate the suspect step in the Plugin Registration Tool, or deactivate the real-time workflow or flow, without unregistering or deleting anything. Pause queued jobs and integrations that would keep writing bad values, list the affected records from audit history, and repair them only after the fix has shipped.

How do we stop bulk updates timing out plug-ins in Dynamics 365?

Keep synchronous steps to validation and values that must be right at save, and move fan-out work into chunked asynchronous processing through a queue with Azure Functions or a paging Power Automate flow. Batch integration requests with ExecuteMultiple or the Multiple messages, respect service protection limits, and remember each row still runs its plug-ins, so make the step fast before making batches bigger.

Which solution is right for your business?

Tell us what you need. A senior consultant replies within one business day with a recommendation - Dynamics 365, Power Platform, or a custom-built CRM - not a sales script.