Google SRE Chapter 6 — Monitoring Distributed Systems

Measuring an error budget, cutting down toil, catching problems early; all of it rests on one layer: monitoring. Chapter 6 is exactly about that, how monitoring should actually work in distributed systems.

Why We Monitor at All

Google lays out several distinct purposes for monitoring: long-term trend analysis, comparing behavior over time, alerting, dashboards, and post-incident debugging. This distinction matters, because a single monitoring system that tries to serve all of these purposes equally well usually ends up serving none of them particularly well.

The Four Golden Signals

This is the most quoted, most practical part of the chapter. Google recommends four core metrics to focus on when monitoring a service.

Latency, how long requests take to process. There's a key nuance here: you need to separate the latency of successful requests from the latency of failed ones, since an error can come back very fast, and that's not actually a good thing.

Traffic, the volume of demand hitting the system; requests per second, transactions per second, that kind of measure.

Errors, the rate of requests that fail.

Saturation, how full the system's resources are, CPU, memory, disk I/O. As a system approaches saturation, performance typically degrades slowly at first, then suddenly.

Google's advice is blunt: if you can only monitor four things, monitor these.

The Philosophy of Alerting: Is It Worth Paging Someone?

The chapter calls out the biggest trap in monitoring systems: alert fatigue. Setting too many alerts, too sensitively, leaves people unable to tell what actually matters from what doesn't. Google's rule here is strict: an alert should only exist for something that genuinely needs human intervention, is urgent, and is actionable. Anything short of that belongs on a dashboard, not in a page.

Three categories are suggested: alerts (need immediate action), tickets (not urgent, but someone needs to look at it), and logging (just needs to be recorded).

Symptoms, Not Causes

Google recommends building alerts around symptoms rather than possible causes. An alert for "users are getting errors" is worth more than one for "disk usage crossed 90%," because the root cause can vary, but the symptom, meaning the user is actually affected, always points to a real problem.

Closing Thoughts

Chapter 6 puts monitoring inside a proper philosophical framework: know what you're watching and why, avoid alert fatigue, and think in terms of symptoms. These principles are exactly what the troubleshooting and incident management chapters later in the book build on.

Through a 2026 Lens: Four Signals Became the Standard, So What Did AI Change?

What stands out most reading this chapter today is how the "Four Golden Signals" idea spawned its own derivatives. Frameworks like RED (Rate, Errors, Duration) and USE (Utilization, Saturation, Errors) are really just adaptations of these same four signals for different kinds of services, request-driven versus resource-driven. Working daily with the LGTM stack (Loki, Grafana, Tempo, Prometheus), I can say without hesitation that these four signals still form the skeleton of pretty much every dashboard out there.

The genuinely interesting tension is between the book's advice that "simple thresholds tend to be more reliable" and today's AI-driven anomaly detection. A decade ago, setting an alert meant picking a fixed threshold: "warn me if CPU crosses 80%." Today, a lot of observability platforms learn a dynamic baseline instead and flag deviations from it as anomalies. On the surface, that looks like a direct challenge to the book's "simple is always better" thesis, but I don't think it actually contradicts it, the definition of "simple" just shifted. The book was warning against complex nested conditionals, since those tend to be fragile and hard to maintain. AI-based anomaly detection doesn't really make the operator write complex conditions either, it just hides that complexity inside the model. So the principle holds, only the tool changed.

One caveat is worth adding: AI-based anomaly detection doesn't always answer the "why" as clearly as the book recommends. The strength of symptom-based alerting was that you could say "users are affected" without even knowing the cause. A model flagging something as "abnormal" doesn't always give you that same clarity, which is why pairing AI-based detection with symptom-based, human-readable alerts is still the healthier approach.

Next time, I'll move on to Chapter 7: The Evolution of Automation at Google.

Thanks for reading.