Fifteen Point-to-Point Integrations You Cannot Maintain: Rationalising Without New Middleware
A decision guide for mid-market Dynamics 365 estates: inventory and rank, stabilise with what you already license, make queues and retries trustworthy, then decide honestly whether middleware is justified.
Before buying middleware, inventory every integration: its trigger, direction, daily volume, owner and failure history, then rank each one by what a day of failure costs the business. Stabilise the top of that list with what you already license: a kill switch per flow, a synthetic smoke test per flow, alerts on silence as well as on errors, and one shared adapter per external API so retries, throttling and token refresh are handled in one place. Add an outbox table with retry, backoff and a circuit breaker per endpoint, and make Azure Service Bus consumers peek-lock and idempotent. Only then test whether fan-out, ordering or volume genuinely justify a broker or iPaaS.
Why do fifteen point-to-point integrations become impossible to maintain?
Nobody designs fifteen point-to-point integrations. They accumulate: a flow to the web shop, a scheduled job to the ERP, a plug-in that calls the address validation service, a file export a supplier collects, a Service Bus queue somebody added for one project. Each was reasonable on the day it was built. Together they form an estate where every connection has its own retry logic or none, its own credentials, its own idea of which system owns the customer record, and its own way of failing.
The symptoms are consistent. A change to one field breaks three integrations nobody knew read it. The daily order sync fails a few times a month and somebody re-runs it by hand. Each integration was built by a different person, so a failure is diagnosed from scratch every time. The instinct is to buy an integration platform and move everything onto it. That is sometimes right, but it is the most expensive first step, and moving fifteen fragile integrations onto a new platform usually produces fifteen fragile integrations on a new platform.
- No inventory: nobody can list every integration, its owner and what it runs as.
- Duplicated handling: six integrations call the same external API, each with different retry and error logic.
- Polling everywhere: jobs that read rows modified since the last run and lose records when a run fails or overlaps.
- No way to pause: stopping a misbehaving integration means turning off a flow and hoping nothing is lost while it is off.
- Failure is discovered by the business: a customer, a warehouse or finance notices before any alert does.
What should the integration inventory record before any platform decision?
The inventory is the single most useful artefact in the whole exercise, and it is worth finishing before any other decision. It is built from the environment itself, not from memory: Power Automate flows and their connections, plug-in steps and service endpoints registered in Dataverse, System Jobs, application users and app registrations, scheduled jobs on servers, and the Azure resources in the subscription. If the estate was left behind by a partner who has gone, our Dynamics 365 project rescue and takeover service covers recovering access and the identities integrations authenticate as first.
| Column | What to record | Why it matters later |
|---|---|---|
| Trigger | Dataverse event, schedule, polling query, webhook, file arrival, queue message or manual run. | Polling and schedules are where silent gaps come from; events are easier to make reliable. |
| Direction and systems | Source, target, and whether it reads, writes or both. | Two integrations writing the same column from different systems is a common cause of data that changes back. |
| Volume | Typical records per day and the peak, such as month end or a promotion. | Volume decides whether throttling, batching or a queue is the real problem. |
| Owner | A business owner who knows what failure costs, and a technical owner who can fix it. | An integration without a business owner cannot be ranked, and one without a technical owner cannot be fixed. |
| Identity | The application user, service principal, service account or named person it runs as, and any secret expiry date. | Expired secrets and departed users stop integrations with no warning. |
| Failure history | Failures over recent months from run history, System Jobs, logs and support tickets, and how each was noticed. | An integration that fails often but is noticed quickly is a different priority from one that failed once silently. |
| Replay and idempotency | Can one record be resent, and does resending create a duplicate? | This decides whether the integration can be stabilised in place or has to be redesigned. |
How do you rank integrations by business impact?
Rank on consequences, not on how irritating an integration is to the person who maintains it. Score each integration on the four questions below with the business owner in the room, then work the list from the top. The daily order sync that finance re-runs by hand usually ranks above the elegant but unimportant marketing sync, and the ranking stops the loudest team setting the order.
| Question | Low | High |
|---|---|---|
| What does one day of failure cost? | An internal report is a day late. | Orders are not shipped, invoices are not raised, or customers act on wrong data. |
| How would anyone notice? | An error alert reaches a named owner the same hour. | Only a customer, a supplier or a month end reconciliation would find it. |
| How often does it fail? | Rarely, and with a clear cause. | Repeatedly, with a different explanation each time. |
| How hard is recovery? | Resend by key, no duplicates. | Manual re-keying, or a replay that creates duplicates. |
What stabilisation buys months without buying a new licence?
Four controls, built with the Power Platform and Azure capabilities most Dynamics 365 tenants already have, remove most of the day to day pain and give you the evidence to decide what to rationalise. None of them requires new middleware. Detecting records that have already gone missing and tracing where they were lost is a separate job, set out step by step in our emergency triage guide for integrations that silently lose records; this page picks up once the bleeding has stopped. The flow engineering underneath these controls, including scopes, retry policies and service accounts, is what our Power Automate consulting delivers.
- A kill switch per integration: a row in a small configuration table that each flow or plug-in reads at run time, so an integration can be paused without turning anything off and without losing work, because pending items stay in the outbox described below. A configuration table is preferable to an environment variable for this, because flows can keep using a cached environment variable value until they are saved or turned off and on again.
- A synthetic smoke test per integration: a scheduled run that pushes a known test transaction, or where that is not possible a harmless authenticated read, through the whole path, including authentication, and records pass or fail. It proves the integration works before real data needs it, and it catches an expired secret at seven in the morning rather than at the first failed order.
- Alerting on silence, not only on error: an integration that normally moves records every hour and has moved none since the morning is an incident even though nothing failed. The check runs under a different identity from the integration it watches, so the same expired credential cannot silence both.
- A shared adapter per external API: one child flow, custom connector plus child flow, or small Azure Function that every integration uses to call that system. Token refresh, timeouts, retry on throttling responses honouring the Retry-After header, error mapping and logging are then written once, so six integrations calling one flaky API stop failing in six different ways.
Why does a daily order sync keep failing against a flaky external API?
Daily batch syncs fail for structural reasons more often than because the external API is unusually bad. A batch that processes hundreds of orders in one run inherits every weakness of the API at once, and one bad record or one timeout can decide the outcome for all of them. Most of the fixes below are design changes rather than new tools.
| What happens | Why | Fix |
|---|---|---|
| The whole run fails because of one order | The batch is all or nothing, so a single mapping error or rejected record stops the loop. | Process per record, record the outcome of each, and park failures for review while the rest continue. |
| It fails at the same time every few days | The run collides with the API maintenance window, a token lifetime or a rate limit reset. | Move the schedule, refresh tokens in the adapter, and spread load rather than sending everything at once. |
| It reports success but orders are missing | Only the first page of results was read, or a condition skipped the write without failing. | Follow pagination explicitly and compare the count processed with the count the source reports. |
| Orders arrive twice after a re-run | The write is a create, not an upsert, and the re-run cannot tell what already arrived. | Write on a shared business key, using an alternate key in Dataverse, so a resend updates rather than duplicates. |
| Throttling errors at month end | Volume peaks hit the API or Dataverse service protection limits. | Honour Retry-After, batch sensibly, and queue work instead of dropping it after the retries run out. |
How does an outbox table with retry, backoff and circuit breaking work?
The outbox pattern separates deciding to send from sending. When a record changes in Dynamics 365, the change and a row describing the message to send are committed together, for example by a synchronous plug-in that creates the outbox row inside the same transaction, so if the save rolls back the outbox row does too. A dispatcher, a scheduled cloud flow at modest volume or an Azure Function at higher volume, picks up pending rows and calls the endpoint through the shared adapter. Nothing is sent from inside the user's save, and nothing is lost because an endpoint happened to be down at that moment.
The outbox can be a Dataverse table or a table in an Azure SQL database you already run. In Dataverse, purge completed rows on a schedule so the table does not consume storage capacity indefinitely.
- Columns: business key, target endpoint, payload or a reference to the source record, status (pending, sent, failed, dead), attempt count, next attempt time, last error and correlation identifier.
- Retry with exponential backoff: each failure pushes the next attempt time further out, with a little random jitter, so a recovering endpoint is not hit by every pending row at once.
- A maximum attempt count: after it, the row moves to dead, an alert is raised and a person decides, instead of the row retrying forever or disappearing.
- A circuit breaker per endpoint: after a run of consecutive failures the dispatcher stops calling that endpoint for a cool-down period, then sends a single probe; rows keep accumulating safely while the circuit is open.
- Idempotent writes at the target: an upsert on the business key or an idempotency key header where the API supports one, because a timeout after a successful call will cause a resend.
- Replay is a status change: setting dead rows back to pending resends exactly those messages, by key, with no date range guesswork.
Why does Azure Service Bus appear to lose orders, and how do you stop it?
Azure Service Bus is durable, and when orders go missing around it the cause is almost always in how messages are sent, received or settled rather than in the service itself. Messages sent from Dataverse through a registered service endpoint are posted by an asynchronous step, so a failed post shows up in System Jobs rather than in the queue. The table covers the patterns we check first.
| What looks like loss | What actually happened | Fix |
|---|---|---|
| The consumer crashed and the message is gone | The receiver used receive-and-delete mode, so the message was removed from the queue as soon as it was received. | Use peek-lock: the message is locked, not removed, and is completed only after the write to the target has been committed. |
| The message sits in the dead-letter queue nobody reads | Processing failed or the lock expired repeatedly, the delivery count passed the maximum delivery count (10 by default) and Service Bus moved it to the dead-letter queue. | Monitor the dead-letter message count per queue and subscription, read the dead-letter reason, fix the cause and resubmit. Dead-lettered messages stay until someone receives them. |
| Messages expired quietly | The message time to live passed before anyone processed it, and dead-lettering on message expiration was not enabled, so the message was discarded. | Set time to live deliberately and enable dead-lettering on expiration, so an expired order is kept for review. |
| The sender succeeded but no subscriber received it | A message sent to a topic with no subscription whose filter matches it is accepted and then discarded. | Review subscription filters, and consider a catch-all subscription you monitor for unrouted messages. |
| The consumer completed a message it did not process | An exception was caught and logged, and the message was completed anyway. | Complete only on success; abandon transient failures so they are retried, and explicitly dead-letter non-retryable ones with a reason. |
| Orders processed twice | A lock expired during slow processing and the message was redelivered, or the sender retried after a timeout. | Keep processing short or renew the lock, make the consumer idempotent on the order key, and set the message identifier to the business key with duplicate detection enabled on the queue. |
| Updates applied out of order | Several consumers processed messages for the same order in parallel. | Use sessions with the order number as the session identifier where order matters, so messages for one order are processed in sequence. |
- Duplicate detection and sessions are chosen when a queue or topic is created and need the Standard or Premium tier, not Basic. Enabling them on an existing entity generally means creating a new one and moving senders and receivers across, so plan it as part of rationalisation rather than as an emergency change.
- Duplicate detection discards a message whose message identifier was already seen within the configured time window. It only works if the sender sets the message identifier to something meaningful, such as the order number plus a version, rather than a random value.
- Duplicate detection protects against a sender resending; it does not protect against a message being redelivered to a consumer after a lock expires. The consumer still has to be idempotent.
- Sessions give ordered, one-at-a-time processing per session identifier, which is right for a sequence of updates to one order and unnecessary overhead for independent messages.
- A peek-lock consumer and an idempotent write together give effectively-once results on top of at-least-once delivery, which is the realistic goal.
What does rationalisation look like once the estate is stable?
With the inventory, the rankings and the stabilisation controls in place, rationalisation is a sequence of small, reversible changes, each proven by the smoke tests and the silence alerts before the next one starts. The aim is fewer integrations, one owner of each piece of data, and one way of doing each thing.
- Agree the system of record for each shared entity, such as customer, product, price and order status, and remove integrations that write the same data in both directions.
- Merge integrations that connect the same pair of systems into one, sharing the adapter and the outbox.
- Replace polling with events where the source supports them: Dataverse triggers and service endpoints on the CRM side, webhooks on the external side.
- Give every integration an alternate key to upsert on, so every integration can be replayed safely.
- Retire integrations the inventory shows nobody depends on, after a period switched off by the kill switch rather than deleted.
- Write a one page contract per integration: fields, direction, keys, volume, owner and what happens on failure.
When is a broker or iPaaS genuinely justified?
Middleware is a real answer to real problems, just not to the problem of integrations nobody has inventoried. Apply the honest test after stabilisation, because by then you know your volumes, your failure causes and which integrations actually matter. If most answers below are no, the stabilised estate is the right architecture for now. If several are yes, start with the Azure services the tenant already has before buying a separate platform, and budget for them honestly: they are billed on consumption, not free.
We recommend the right solution - whether that's Microsoft Dynamics 365, Power Platform, or a custom-built CRM. Some businesses need the Microsoft ecosystem. Others need full control without licensing. We deliver both. If the review shows the CRM itself no longer fits, for example because licensing does not suit the users the integrations exist to serve, our custom CRM development builds CRM on React, Node.js, PostgreSQL and .NET with the integrations designed in from the start.
| Question | If yes | Azure option often already available |
|---|---|---|
| Does one business event need to reach several systems? | Fan-out through a topic is simpler than several point-to-point calls. | Service Bus topics and subscriptions, fed by a Dataverse service endpoint. |
| Must updates for the same record be processed strictly in order? | Point-to-point flows running in parallel cannot guarantee it. | Service Bus sessions. |
| Is volume beyond what scheduled flows and service protection limits handle comfortably? | A queue with scalable consumers absorbs peaks. | Service Bus queues with Azure Functions consumers. |
| Do many different teams need governed access to the same APIs? | A managed API layer with policies and keys pays for itself. | Azure API Management. |
| Is there someone to operate it? | Middleware needs an owner, monitoring and deployment discipline. | Only buy or build what an existing team will own. |
Should the CRM behind these integrations stay on Dynamics 365, move to Power Platform or be custom-built?
Can afford licensing and want the Microsoft ecosystem
Dynamics 365
Microsoft 365, Teams and Outlook integration, a mature partner ecosystem, Copilot, and apps for sales, service and field operations that are configured rather than built.
Need full control and zero licensing
Custom CRM
A CRM built on React, Node.js, PostgreSQL or .NET that you own outright: your data model, your hosting, no per-user subscription, and features shaped exactly to your process.
Not sure which fits
We help you decide
A short discovery weighs licensing budget, process complexity, integrations and long-term ownership, then recommends one path. We deliver both, so the recommendation has no reason to lean.
How does a rationalisation engagement run on a mid-market budget?
Global systems integrators sell this as an enterprise integration programme. A mid-market organization with a procurement freeze usually needs something smaller: a fixed scope inventory and ranking, stabilisation of the integrations at the top of the list, and a written recommendation on middleware that is allowed to say no. Solzet works on the CRM side and the integration layer, Dynamics 365 Customer Engagement, Dataverse, Power Automate, plug-ins, service endpoints and the Azure components around them. We do not implement or modify Dynamics 365 Finance, Business Central, Finance and Operations or any other ERP; where an ERP interface needs changing we specify the contract and work with the team that owns it.
Work is delivered remotely from Yerevan by senior consultants and full-stack developers with 8+ years of Dynamics 365 and Power Platform delivery. If you need developers added to your own team for the build, that is how our Dynamics 365 developer service works, and if the integrations sit inside an estate whose partner has gone, start with the project rescue and takeover service.
- Inventory and ranking: the table above, built from the environment, reviewed with business owners.
- Stabilisation: kill switches, smoke tests, silence alerts and shared adapters for the highest ranked integrations.
- Resilience: outbox, backoff, circuit breaking and idempotent Service Bus consumers where the ranking justifies them.
- Rationalisation: merges, retirements and system of record decisions, one reversible change at a time.
- Recommendation: a written answer on whether a broker or iPaaS is justified, and if so which, with the reasons.
What do people ask us?
We have too many point-to-point integrations and no middleware budget. Where do we start?
Start with an inventory of every integration: trigger, direction, volume, owner, the identity it runs as and its failure history. Rank them by what a day of failure costs the business, then stabilise the top of the list with kill switches, synthetic smoke tests, alerts on silence and a shared adapter per external API. Most of that uses capabilities a Dynamics 365 tenant already has.
Why does our daily order sync fail against an external API?
Usually because the batch is all or nothing, reads only the first page of results, retries without honouring throttling responses, or creates rather than upserts so a re-run duplicates orders. Process per record, follow pagination, handle retries and token refresh in one shared adapter, and write on a business key so resends are safe.
Can Azure Service Bus lose messages?
Service Bus is durable; apparent loss almost always comes from how it is used. Common causes are receive-and-delete mode with a crashing consumer, messages that expired without dead-lettering on expiration enabled, topics with no matching subscription, consumers that complete messages after swallowing an error, and dead-letter queues nobody monitors. Use peek-lock, idempotent consumers and dead-letter monitoring.
What is the difference between duplicate detection and an idempotent consumer?
Duplicate detection makes Service Bus discard a message whose message identifier it has already seen within a time window, which protects against a sender resending. An idempotent consumer makes processing the same message twice harmless, which also covers redelivery after a lock expires. You normally want both, with the message identifier set to a business key.
What is an outbox table and do we need one?
An outbox table records each message to send in the same transaction as the business change, and a dispatcher sends it later with retry, backoff, a maximum attempt count and a circuit breaker per endpoint. You need one for integrations where losing a message is expensive and the endpoint is not always available, which is most order and invoice integrations.
When should we buy an iPaaS or integration broker?
When one event must reach several systems, when strict ordering or high volume is a genuine requirement, when many teams need governed access to the same APIs, and when someone will operate the platform. Test that after stabilising the estate, and consider Azure Service Bus, Azure Functions and API Management already available in your tenant before a separate product.
Our integrations are already losing records. Is this the right guide?
Contain the loss first. Our emergency triage guide for Dynamics 365 integrations that silently lose records covers detecting the gap, proving where records were lost and replaying them safely. This guide is what to do once the immediate damage is contained.
Does Solzet change the ERP side of these integrations?
No. Solzet works on Dynamics 365 Customer Engagement, Dataverse, Power Automate, plug-ins and the Azure integration components. We do not implement or modify Dynamics 365 Finance, Business Central, Finance and Operations or other ERP systems; we specify the integration contract and work alongside the team that owns the ERP.
Where should you go next?
Integration emergency triage
Detect records an integration is silently losing, prove where they were lost and contain it within a day.
Dynamics 365 project rescue and takeover
Taking over an estate whose partner has gone, including the identities integrations run as.
Power Automate consulting
Cloud flows and desktop RPA engineered with scopes, retry policies and service accounts.
Hire a Dynamics 365 developer
Senior developers for Dataverse, plug-ins and integrations, added to your team.
Duplicate customer data cleanup
Fixing the duplicates that overlapping integrations leave behind without breaking reports.
Custom CRM Development
CRM on React, Node.js, PostgreSQL and .NET for organizations that need full control without Microsoft licensing.
Which solution is right for your business?
Tell us what you need. A senior consultant replies within one business day with a recommendation - Dynamics 365, Power Platform, or a custom-built CRM - not a sales script.