# Site Reliability Engineering (SRE) Site Reliability Engineering (SRE) is Google's way of running production systems with software engineers. Benjamin Treynor Sloss, who started it, sums it up as "what happens when you ask a software engineer to design an operations team". He joined Google in 2003 and was put in charge of a "Production Team" of seven engineers, which grew into Google's SRE organization (more than 1,000 SREs by 2016). Google published the practices in *Site Reliability Engineering* (O'Reilly, 2016) and *The Site Reliability Workbook*, both free to read at sre.google. ## Core ideas - **SLIs and SLOs.** A service level indicator (SLI) is a carefully defined measure of the service: request latency, error rate, throughput. A service level objective (SLO) is the target for an SLI, e.g., average search latency under 100 ms. An SLA is the agreement with users about what happens when objectives are missed - **100% is the wrong target.** Users can't tell 100% from 99.999% availability, because their laptop, Wi-Fi and ISP fail more often than that. Every extra nine costs more and slows change - **Error budgets.** The gap between the SLO and 100% is a budget for failure. With an SLO of 99.999% of queries per quarter, the budget is 0.001%; an incident that fails 0.0002% of queries spends 20% of it. While budget remains, teams ship. When it's gone, releases stop (in Google's example, for the rest of the quarter). The book says this removes the politics from the negotiation between SREs and product developers - **Toil.** Operational work that is manual, repetitive, automatable, tactical, without lasting value, and growing linearly with the service. Google's SRE teams aim to keep toil below 50% of each engineer's time and spend the rest on engineering that removes it - **Blameless postmortems.** After an incident, the write-up looks for contributing causes without blaming any person or team, assuming everyone acted in good faith with the information they had. The practice comes from healthcare and aviation. Where people get blamed, they stop reporting problems - **Handling overload.** The book has whole chapters on [[Load Shedding]], [[Cascading Failures]] and [[Retry pattern|retry budgets]] ([[Exponential Backoff]]) ## SRE and DevOps The Workbook describes [[DevOps]] as a philosophy and SRE as a concrete job role plus a set of practices that implement much of it: "class SRE implements interface DevOps". Both rely on measurement, automation and blameless postmortems. The difference is mostly in how concrete SRE is (SLOs, error budget policies, toil caps). ## Limits SRE grew out of Google's scale, hiring and management culture. The book itself notes that a rule like freezing releases when the error budget is spent "might not be embraced by a product development team unless mandated by their management". Renaming an operations team to SRE without SLOs, an agreed error budget policy and time for engineering work changes the title, not the practice. ## References - Benjamin Treynor Sloss, "Introduction", in Site Reliability Engineering (Google, O'Reilly, 2016): https://sre.google/sre-book/introduction/ - "Service Level Objectives" (SRE book, ch. 4): https://sre.google/sre-book/service-level-objectives/ - "Embracing Risk" (SRE book, ch. 3, error budgets): https://sre.google/sre-book/embracing-risk/ - "Eliminating Toil" (SRE book, ch. 5): https://sre.google/sre-book/eliminating-toil/ - "Postmortem Culture: Learning from Failure" (SRE book, ch. 15): https://sre.google/sre-book/postmortem-culture/ - Niall Richard Murphy, Liz Fong-Jones, Betsy Beyer et al., "How SRE Relates to DevOps", in The Site Reliability Workbook: https://sre.google/workbook/how-sre-relates/ - Wikipedia, Site reliability engineering: https://en.wikipedia.org/wiki/Site_reliability_engineering ## Related - [[DevOps]] - [[Mean Time To Recovery (MTTR)]] - [[Cascading Failures]] - [[Load Shedding]] - [[Exponential Backoff]] - [[Chaos Engineering]] - [[Fault Tolerance]] - [[Pre-mortem]] - [[Google]]