Monitoring and Observability

Metrics

Metrics are quantitative measurements that give insight into the performance, health and behaviour of a system. Latency, throughput, error rates and resource utilization are the usual ones.

Collecting and analysing them shows where the bottlenecks are and where problems are building. Prometheus, Graphite and InfluxDB collect, store and query them.

Metrics answer how is the system doing? They are aggregate numbers, which is their strength and their limit. A metric tells you the error rate went up. It does not tell you which request failed or why.

Distributed Tracing

Distributed tracing tracks and analyses a request as it flows through the system, so you can see its end-to-end performance and find which component is responsible for a problem.

Jaeger, Zipkin and OpenTelemetry implement it.

Tracing answers what happened to this one request? That is the question metrics cannot answer. When an engineer needs to see the end-to-end path of one slow request across many services, tracing is the component that follows that single request across services and locates the slow one. Metrics would only show that something is slow on average, and logs would mean stitching together entries from every service by hand.

Logging

Logs are records of events or messages generated by the components of a system, giving a detailed view of what actually happened.

Their value comes from being collected centrally. Logs scattered across fifty machines are nearly useless, while the same logs aggregated in one place are how most problems get diagnosed. The ELK stack (Elasticsearch, Logstash and Kibana) and Graylog do the aggregating.

Logs answer what exactly happened, in detail?

Alerting and Anomaly Detection

Alerting watches the system for unusual behaviour or performance problems and notifies the right team when one occurs.

It works two ways: alerts fire on predefined thresholds that someone chose in advance, or machine learning detects behaviour that is unusual compared to normal patterns. The first catches what you expected. The second catches what you did not.

Either way the point is to act before users are affected. Grafana, PagerDuty and Sensu do this.

Alerting answers when should a human look?

Visualization and Dashboards

Dashboards aggregate metrics, traces and logs into a view that can be read at a glance, which is what makes a decision possible rather than an investigation. Grafana, Kibana and Datadog build them.

Dashboards answer what is the state of everything, right now?

Choosing the Right One

💡 A good answer to "how would you debug this?" names the tool that fits the question. A rising p99 with a healthy average is a tracing problem, not a metrics problem, because the aggregate is hiding the requests you care about. Saying that out loud is worth more than listing five tools.

Key takeaway: Metrics are quantitative measurements such as latency, throughput, error rates and resource utilization, good for aggregates. Distributed tracing follows one request across services, which is the only way to locate where a single slow request lost its time. Logs record detailed events and are worth aggregating centrally. Alerting fires on predefined thresholds or on machine-learned anomalies. Dashboards pull all of it into one view.

The next lesson, Resilience and Error Handling, covers what to do once monitoring tells you something has broken.