iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A distributed system survives a network split by deciding which parts may keep making authoritative progress—and which must wait. In a quorum-based system such as etcd, the majority side can continue while the minority stops accepting consensus-dependent writes. That trade-off protects a single authoritative history; it does not keep every side independently writable.
1. Let quorum decide which side can commit
A network partition divides cluster members into groups that cannot communicate. In etcd, the group with a majority of the configured members remains available; the minority is unavailable. Majority is measured against the cluster’s configured membership, not just the number of nodes that happen to be able to see one another.
If the leader is isolated on the minority side, it steps down and the majority elects a new leader. When communication returns, the minority recognizes the majority’s leader and recovers its state. This keeps the cluster from treating two disconnected groups as equally authoritative.
The cost is availability: if the cluster cannot reach a majority, it cannot accept writes that require consensus. If the majority cannot return, operators need disaster recovery. Quorum therefore answers a deliberate design question: is it more important to preserve one authoritative write history, or to allow isolated parts of the system to accept writes and reconcile them later?
#1 Best Overall
2. Put replicas in independent failure zones—and protect the endpoints
Replicas spread across independent zones can reduce exposure to a failure concentrated in one zone, but placement alone does not guarantee that clients can reach a healthy control plane. Kubernetes recommends selecting at least three failure zones and replicating each control-plane component across at least three zones when availability is important. Its topology-spread constraints can help distribute Pods.
Kubernetes states that “Kubernetes does not provide cross-zone resilience for the API server endpoints.” The API endpoint therefore needs its own resilience design, such as DNS round-robin, SRV records, or a third-party load balancer with health checks.
Rank #2
Evaluate whether both the replicas and the communication paths between them would remain reachable during the failure the design is meant to withstand. A multi-zone deployment can still have a vulnerable endpoint, network, storage layer, or correlated failure. Zone placement also does not automatically make a network plugin zone-aware; check the documentation for the cloud provider and network plugin used in the deployment.
3. Design clients for elections, timeouts, and ambiguous outcomes
A cluster may preserve its consistency guarantees while applications experience interruptions. During RabbitMQ quorum-queue leader changes, publisher confirms can be delayed or rejected in some scenarios, and an application may need to publish again later. Consumer registration and polling require a reachable leader and can block until an election completes or time out. Some operations may be buffered and replayed against the new leader.
Rank #3
Client behavior should distinguish a confirmed success from an unknown outcome. A timeout does not necessarily prove that the operation failed: the request may have reached the service even if its acknowledgement did not reach the client. Retrying such an operation can create duplicate effects unless it is safe to repeat or protected against duplicates.
- Set timeouts appropriate to the service and allow for leader election and temporary loss of connectivity.
- Retry only operations that are safe to repeat, or use application-level protections against duplicate effects.
- Handle delayed or rejected acknowledgements explicitly rather than treating every interruption as a confirmed failure.
- For linearizable reads using the etcd Raft library, account for its quorum checks; its lease-based linearizable reads rely on the clocks of machines in the Raft group.
These are client and consistency considerations, not a promise that an arbitrary retry is safe. RabbitMQ’s documented outcomes describe its operations; application semantics determine whether repeating a particular action is acceptable.
Rank #4
4. Plan for reconnection, catch-up, and loss of quorum
Recovery after a partition is more than reconnecting a link. In etcd, a minority member recognizes the majority’s leader and recovers its state after connectivity returns. RabbitMQ documents that a reconnected Raft member discovers the elected leader and receives missing log entries. After a long interruption, catch-up can involve substantial data, so treat that member as temporarily unavailable while it catches up.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsFor Kubernetes-backed etcd, the official operations guidance recommends periodic backups and a multi-node production cluster, and recommends five members for production. Confirm the guidance for the Kubernetes and etcd versions and operating environment in use before making operational changes. If a majority of etcd members have permanently failed, Kubernetes cannot change the currently stored cluster state until the cluster is recovered.
- Keep periodic backups and know how to restore them.
- Plan how operators will decide whether quorum can return or disaster recovery is required.
- Allow for a recovering member to remain unavailable while it receives missing log entries.
- Use a failure-tolerance figure only for the system and configuration it describes. RabbitMQ’s documented table says five Raft members tolerate two member failures; that is not a universal guarantee for distributed systems.
How to think about a network split
Evaluate a design by asking which partition can make progress, what consistency guarantees remain in force, how many and what kinds of failures the configuration tolerates, what clients see during elections, and how members recover and catch up. Quorum-based consensus makes behavior predictable when a majority remains online, at the cost of stopping consensus-dependent operations when it does not. Multi-zone placement can reduce exposure to some failures, but it does not replace endpoint resilience, client handling, backups, or a recovery plan.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
