National Infrastructure · 2025

Isolated sites lost database access, and the cause was cluster quorum.

A real-time control platform lost database access at isolated sites. Structured discovery traced it to quorum vote distribution, surfaced a separate backup exposure, and set out what the technology could reliably deliver.

Client
Control platform provider
Sector
National Infrastructure
Services
SQL Server High Availability, Discovery & Assessment
Year
2025
Timeline
Discovery to design
Read time
5 min read

4

SQL Server Enterprise instances in one stretched cluster

3

Target-state options assessed to the same standard

275 GB

Unmanaged transaction log found during discovery

01 · The challenge

What we walked into

A technology provider runs a real-time control platform across three geographically separated control centres, on four SQL Server Enterprise instances in a single Windows Server Failover Cluster and a single Always On availability group. Whenever a remote centre lost its network connection to the primary site, its operators lost all access to the database their systems depend on.

The stated requirement was straightforward to describe and considerably harder to deliver. Each control centre had to stay usable during a network isolation, including the ability to write, with the primary site remaining authoritative and reconciling the isolated site on reconnection. The working assumption pointed at the application and the network. The evidence pointed somewhere more specific.

02 · Our approach

How we got there

  1. 01

    Structured discovery

    We went and looked at all four instances ourselves, from availability group configuration and cluster quorum to listener and subnet registration, Active Directory topology, inter-site latency, and the databases themselves.

    Evidence from the estate

  2. 02

    Diagnosis

    The availability group was healthy. The failure was quorum arithmetic, with five cluster votes and the witness hosted at the primary site, so an isolated remote centre fell below majority and its cluster service stopped by design.

    Root cause

  3. 03

    The second finding

    Backup preference targeted a secondary replica that cannot run full backups, log jobs were failing, and one database had no backup history at all. A roughly 275 GB transaction log could not truncate. We raised it as a live data-loss exposure, independent of the availability work.

    Raised as a priority

  4. 04

    Options and recommendation

    We assessed three approaches to the same standard, verified vendor limitations against documentation, and recommended immediate backup remediation and witness relocation while gating the target state on confirmations only the application owner can give.

    Decision-ready

03 · The outcome

What changed

The customer received an evidenced diagnosis, an immediate reduction in live exposure, and a set of target-state options with their real limitations stated up front. The choice stayed with them, made against a complete picture of what the technology can and cannot do.

Evidence over assumption

The cause was demonstrable from the estate's own event history, spanning roughly ten months of production operation.

Hidden exposure closed

The backup failure carried more immediate risk than the availability problem that prompted the engagement.

Options with their limits stated

Two of the three candidate designs could not meet the requirement as written. We set out why, with vendor documentation behind every conclusion.

Technologies

  • SQL Server Enterprise
  • Always On Availability Groups
  • Windows Server Failover Clustering

A note on confidentiality

Much of our work takes place in regulated and security-sensitive sectors, where the details of a client's environment are confidential by default. In line with our commitments to those clients, identifying details in this case study have been anonymised or generalised, while the work, the decisions, and the outcomes are described as delivered.

If you would like to know more about this engagement, we are happy to discuss it further and can share additional information on request.

Next case study

Six hundred people switched tenants over a single weekend

Technology Services · 2025 · 4 min read

600+

Users migrated with mail and OneDrive intact