# Fault Tolerance Fault tolerance is the ability of a system to keep working correctly when some of its components fail. A fault-tolerant system masks the error: users don't notice that a disk died, a server crashed or a message got lost. The first known fault-tolerant computer, SAPO (Czechoslovakia, 1951, Antonín Svoboda), already used voting between redundant units to catch memory errors. ## Fault tolerance, high availability and resilience The three terms overlap, and people use them loosely. One useful split: - **Fault tolerance**: faults are handled without degradation or downtime. Users see nothing - **Resilience**: the system keeps serving, but with some interruption or reduced quality ("graceful degradation"). Users may see a slower page or a missing feature - **High availability**: a target for uptime (e.g., 99.99%), usually expressed as an SLO. Fault tolerance is one way to reach it; fast detection and recovery is another ## How systems tolerate faults - **Redundancy and failover**: run several identical instances and switch to a healthy one when one fails - **Replication with voting**: send the work to several replicas and take the answer the majority agrees on - **Diversity**: use different implementations of the same specification, so that one bug doesn't hit every replica at once - **Isolation**: contain the fault where it happens, with [[Timeouts (software)|timeouts]], a [[Circuit Breaker]] and [[Bulkheads]], so it doesn't turn into [[Cascading Failures]] - **Clean failure**: components that [[Fail Fast]] and report errors clearly are easier for the rest of the system to route around than components that limp along with bad state - **Testing**: [[Chaos Engineering]] injects faults on purpose to check that the redundancy actually works ## Byzantine faults The hardest case is a component that doesn't simply stop, but keeps running and gives wrong or inconsistent answers to different parts of the system. [[Leslie Lamport]], Robert Shostak and Marshall Pease described it in 1982 as "The Byzantine Generals Problem". Their result: agreement is possible only if fewer than one third of the participants are faulty. In 1999, Miguel Castro and [[Barbara Liskov]] published Practical Byzantine Fault Tolerance (PBFT), which made it fast enough for real replicated systems. Blockchains and some multi-agent systems use BFT-style consensus. ## Limits Redundancy costs money: spare capacity sits idle until something fails. It also adds complexity, and complexity creates new faults (failover logic is code too, and it runs rarely). Replicas that share a bug, a configuration or a deployment fail together, which is why diversity and staged rollouts matter as much as the number of copies. ## References - Wikipedia, Fault tolerance: https://en.wikipedia.org/wiki/Fault_tolerance - Wikipedia, Byzantine fault: https://en.wikipedia.org/wiki/Byzantine_fault - Leslie Lamport, Robert Shostak, Marshall Pease, "The Byzantine Generals Problem", ACM TOPLAS 4(3), 1982: https://www.microsoft.com/en-us/research/publication/byzantine-generals-problem/ - Miguel Castro and Barbara Liskov, "Practical Byzantine Fault Tolerance", OSDI 1999 - https://concepts.dsebastien.net/concept/fault-tolerance/ ## Related - [[Circuit Breaker]] - [[Bulkheads]] - [[Timeouts (software)]] - [[Cascading Failures]] - [[Chaos Engineering]] - [[Fail Fast]] - [[Site Reliability Engineering (SRE)]] - [[Mean Time To Recovery (MTTR)]] - [[Building graceful degradation into your PKM system]] - [[Leslie Lamport]] - [[Barbara Liskov]] - [[Multi-Agent System (MAS)]] - [[Load Shedding]] - [[Exponential Backoff]] - [[Paxos algorithm]] - [[Retry pattern]]