# Chaos Engineering Chaos engineering is "the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production" (Principles of Chaos Engineering). In practice, you cause failures on purpose (kill a server, cut a network link, inject latency) under controlled conditions and check that the system keeps working. ## How an experiment works The Principles describe four steps: 1. Define the **steady state** as a measurable output that shows normal behavior: throughput, error rates, latency percentiles 2. Hypothesize that the steady state will hold in both a control group and an experimental group 3. Introduce variables that mirror real events: servers crashing, disks failing, network connections cut, traffic spikes 4. Try to disprove the hypothesis by looking for a difference between the two groups The harder it is to disturb the steady state, the more confidence you can have in the system. If a weakness shows up, you have something to fix before it causes an outage. The advanced principles add: vary real-world events, run experiments in production, automate them to run continuously, and minimize the blast radius. ## Origin - Amazon ran "GameDay" exercises from 2003 (Jesse Robbins), purposely creating major failures on a regular basis; Google started DiRT (Disaster Recovery Testing) in 2006 - Netflix built **Chaos Monkey** while moving to AWS (Wikipedia dates it to 2011). It randomly terminates instances in production, so engineers have to build services that survive it. Netflix released the code in 2012 under the Apache 2.0 license, and the Simian Army extended the idea up to Chaos Kong, which takes out a whole AWS region - The Principles of Chaos Engineering (principlesofchaos.org) and a 2016 IEEE Software paper by Netflix engineers (Basiri et al.) formalized the practice. [[Michael Nygard]] gives the topic its own chapter in the second edition of [[Release It! (Book)|Release It!]] (2018) ## Why it matters In distributed systems, the failures that hurt come from interactions: fallbacks that don't work when a service is unavailable, [[Retry pattern|retry storms]] caused by badly tuned [[Timeouts (software)|timeouts]], [[Cascading Failures]] when a single point of failure crashes. These examples come from the Principles. Unit tests don't exercise any of them. Experiments check whether the [[Circuit Breaker]], the timeouts and the fallbacks behave the way the design says. It's the experimental counterpart of a [[Pre-mortem]]: the pre-mortem imagines the failure on paper, chaos engineering causes a small one on purpose. The idea is also close to what Nassim Taleb calls antifragility in [[Antifragile (book)|Antifragile]]: frequent small stressors make a system stronger against large ones. ## Limits - You need good monitoring first. Without a measurable steady state there's nothing to compare - Experiments in production can hurt real customers, hence the rule to minimize and contain the blast radius - It finds weaknesses but doesn't fix them: the value depends on the team acting on the results ## References - Principles of Chaos Engineering: https://principlesofchaos.org/ - Netflix Chaos Monkey repository: https://github.com/Netflix/chaosmonkey - Ali Basiri et al., "Chaos Engineering", IEEE Software 33(3), 2016: https://arxiv.org/abs/1702.05843 - Casey Rosenthal and Nora Jones, Chaos Engineering, O'Reilly, 2020 - Tom Limoncelli, "Resilience Engineering: Learning to Embrace Failure", ACM Queue, 2012 (GameDay and DiRT): https://queue.acm.org/detail.cfm?id=2371297 - Michael Nygard, Release It! Second Edition, Pragmatic Bookshelf, 2018 (Chaos Engineering chapter): https://pragprog.com/titles/mnee2/release-it-second-edition/ - Wikipedia, Chaos engineering: https://en.wikipedia.org/wiki/Chaos_engineering - https://concepts.dsebastien.net/concept/chaos-engineering/ ## Related - [[Cascading Failures]] - [[Circuit Breaker]] - [[Backpressure]] - [[Pre-mortem]] - [[Antifragile (book)]] - [[DevOps]] - [[Continuous Deployment (CD)]] - [[Release It! (Book)]] - [[Michael Nygard]] - [[Fault Tolerance]] - [[Site Reliability Engineering (SRE)]]