Emergency Stabilisation When a Business-Critical System Is Failing Daily

When a business-critical system crashes every week, staff are afraid to log in and customers are leaving, the first 48 hours matter more than the eventual fix. Do five things in this order. Appoint one decision-maker with authority to freeze and approve changes. Take a backup and verify that it actually restores before anyone touches anything else. Stop the changes that are making it worse. Instrument enough to see what really fails instead of guessing. Put a manual fallback in place for the one process the business cannot lose. Then triage whether this needs stabilisation, targeted remediation or replacement, using evidence gathered now, so the decision is made on data rather than panic.

Why do rushed fixes make a failing system worse?

A system that fails daily creates enormous pressure to do something, and the things people do under that pressure are usually the ones that deepen the crisis. Someone applies a hotfix directly to production. Someone else restarts the server, which clears the logs that showed why it crashed. A vendor rolls back a release without checking whether the database has moved on since. Three people change three things on the same afternoon, and when the next crash comes nobody can say which change caused it.

The result is a system that is less stable, with less evidence, and a team that has lost confidence in every fix. The steps below are ordered to prevent that. None of them repairs the system. Together they stop it getting worse, protect what cannot be recreated and make the real cause visible. They apply whether the system is a custom-built application, a legacy line-of-business system or a Dynamics 365 environment.

Who makes decisions in the first 48 hours?

One person, named in writing on the first day. Not a steering group and not whoever shouts loudest. That person needs the authority to freeze changes, approve every change that is made, decide when to switch to the manual fallback and speak for the organisation to customers and suppliers.

• Keep one channel of record where every decision and every change is written down with a time.
• Send a short written brief at a fixed time each day: what failed, what was found, what changed and what happens next.
• Tell staff plainly what is safe to use, what is not and what to do instead. Staff who are afraid to log in are usually afraid of losing their work, and a clear instruction removes that fear faster than any fix.
• Agree what customers are told, by whom, so account managers are not improvising different stories.

Why must a verified backup come before anything else?

Because every other step carries some risk, and the backup is what makes that risk recoverable. It must be verified, because an unverified backup is only a hope. Many crises get worse when a team discovers during an attempted recovery that the backup job had been failing for weeks, excluded the file store or could not be restored without a password that left with a former employee.

For a custom or legacy system, a usable backup covers more than the database:

• a full database backup taken now, not last night's scheduled job;
• uploaded files and documents held outside the database;
• the deployed application code and the exact version running in production;
• configuration, environment settings and an inventory of the secrets, certificates and service accounts the system depends on, stored securely.

Then prove it. Restore to a separate server or environment, confirm the application starts, and compare record counts on the tables that matter against production. Store a copy outside the system and outside the hosting account it runs in.

For Dynamics 365, platform backups and their retention window work differently, and the exact steps, including exporting data and configuration you hold yourself, are set out in our Dynamics 365 rescue guide rather than repeated here.

How do you stop the changes that are making it worse?

Freeze all change to production, with the decision-maker as the only person who can approve an exception. Then look at what changed before the failures started, because in a system that ran acceptably until recently, a change is the most likely cause.

• Releases and hotfixes to the application, including ones described as small.
• Changes to the database: new indexes, schema changes, bulk imports or a sudden jump in data volume.
• New or changed integrations, especially anything that now sends far more traffic than before.
• Infrastructure changes: operating system or runtime patches, hosting moves, resized servers, firewall or network changes.
• Expired or rotated certificates, passwords, API keys and licences.
• Scheduled jobs added or moved so they now overlap with peak use.

Put the list in date order next to the timeline of failures. Where a change lines up with the start of the crashes, consider reversing it, but only after the backup is verified and only if reversing it does not break data written since. Where an automated process is actively damaging data, switch it off rather than deleting it, and record what was switched off and when.

How do you see what actually fails instead of guessing?

Instrument before you fix. A team that guesses fixes the loudest symptom, and the system fails again the next week for the original reason.

For the first 48 hours, the aim is enough visibility to connect each failure to a cause, not a monitoring project:

• An external check that tests the system every few minutes the way a user would, so you know exactly when it fails and recovers.
• Application error logs with timestamps, kept and copied somewhere safe before anyone restarts anything.
• Server and database resource readings over time: CPU, memory, disk space, database connections, long-running queries and locks.
• The run history and error logs of every integration and scheduled job.
• A simple crash log kept by the people using the system: the time, what they were doing and what they saw.

Most daily failures in business systems trace back to a short list of causes: resources running out as data or users grow, database locking or slow queries that stall everything behind them, an integration or job flooding the system at a particular time, an expired certificate or credential, a recent change, or a dependency such as hosting or a third-party service failing. Lining up the crash log against the resource and job timelines usually narrows it to one or two within days. When the failures cannot be reproduced on demand, the method is set out in our guide to diagnosing intermittent failures nobody can reproduce, written for Dynamics 365 but applicable in principle to any system.

Which process gets a manual fallback, and how should it run?

Only one, at first: the process whose loss costs the most per day. Taking orders, dispatching jobs, issuing invoices or answering customers are typical candidates. A fallback for everything is too much to run and too much to reload later.

• Write the fallback down on one page: who does it, with what, and when it is switched on and off.
• Capture data in a structure that matches the system, such as a shared spreadsheet with column headings copied from the system fields, so it can be loaded back rather than retyped.
• Give every entry a date, a person and a reference, and keep one copy in one place, not one per team.
• Decide in advance how entries return to the system once it is stable, and who reconciles them.
• Log the hours the fallback costs each week. That figure becomes central evidence for the decision below.

How do you decide between stabilisation, targeted remediation and replacement?

After the first 48 hours, and usually after the first week or two of evidence, the question changes from how to stop the bleeding to what kind of problem this is. There are three honest answers.

Stabilisation is enough when the design is broadly sound and the failures come from operational causes: capacity, configuration, an expired credential, a runaway job or one bad change. The work is fixing those causes, adding monitoring and a safe release process, and the system continues.

Targeted remediation fits when the failures concentrate in a few components: one integration, one module, a set of slow queries or a layer of customisations built badly. Those components are rewritten or replaced one at a time while the rest of the system runs.

Replacement is justified only when the evidence shows the platform itself cannot be made safe: the technology is out of support and cannot be patched, nobody can read or safely change the code, the data model cannot represent how the business works, or the same failures return after every fix. Even then, replacement is phased, with the stabilised system kept running until the new one has proved itself.

Where the triage points towards replacement, choose the platform on requirements rather than in reaction to the crisis. The right answer may be Dynamics 365 and the Power Platform, or a custom-built system without Microsoft licensing, and that choice deserves the same evidence as the triage. Where the original developers are gone, our guide to migrating a legacy system with no original developers covers extracting what the old system knows before it is switched off.

What evidence should you gather now so the decision is not made in panic?

The triage is only as good as the evidence behind it, and much of that evidence disappears if nobody collects it during the crisis. Logs rotate, run history expires and memories fade.

• The failure timeline: every outage and serious error with start time, end time and impact.
• The change timeline from the freeze step, lined up against the failures.
• The instrument readings and error logs, preserved outside the system.
• Business impact in the organisation's own terms: orders or jobs affected, customers who complained or left, and the weekly hours the fallback costs.
• A dependency map: what the system connects to, what depends on it and which account each connection uses.
• The people map: who understands which part of the system, and who has already left.
• Support status: versions in use, whether they are still supported and the state of any vendor contract.

With that evidence, the stabilise, remediate or replace decision becomes a comparison of costs and risks that a board can follow, rather than an argument between the loudest voices.

Where does the path differ for Dynamics 365, custom and legacy systems?

The first 48 hours are the same on any platform. What follows depends on what the system is.

• Dynamics 365 and Power Platform: environment lockdown, sandbox copies, vendor takeover and the structured stabilisation week are covered on our Dynamics 365 project rescue and takeover service page, and the do-it-yourself crisis steps are in the rescue services guide.
• Integrations silently losing records: the detect, prove, contain method is on our integration emergency triage page.
• A failing customer service operation on a platform that cannot be replaced this year: see stabilising customer service on a legacy platform.
• Custom-built systems: Solzet takes over, stabilises and remediates custom CRM and line-of-business applications built on React, Node.js, PostgreSQL and .NET, and builds replacements where the triage points there, through our custom CRM development service.
• Legacy systems on other technologies: the first 48 hours and the evidence still apply, and we help with triage, data extraction and the replacement decision.

Solzet's senior consultants and full-stack developers bring 8+ years in Dynamics 365 Customer Engagement, Power Platform and custom development, delivering remotely from Yerevan, Armenia, directly or white-label for Microsoft partners.

What should you do in the next 48 hours?

1. Name one decision-maker in writing and set up one channel of record and a daily written brief.
2. Take a full backup of the database, files, code and configuration, and restore it somewhere separate to prove it works.
3. Freeze all changes to production and list every change made before the failures began.
4. Switch off, without deleting, anything actively damaging data, and log what you switched off.
5. Start an external availability check, preserve the error logs and record resource usage over time.
6. Put a documented manual fallback behind the one process the business cannot lose, and log what it costs.
7. Start the failure timeline, business impact record and dependency map that the triage decision will need.