Chaos Engineering for Beginners: How to Break Your System Before Production Does

What This Blog Covers

• What chaos engineering is and why it matters
• The four-step experiment loop
• Five experiments every team should start with
• Tools for running chaos experiments
• How to discuss chaos engineering in system design interviews

It is 3 AM. Your payment service is down.

The on-call engineer checks the dashboard: the database is healthy, the application servers are running, CPU and memory look fine.

Everything looks green. But the service is not responding.

After 45 minutes of investigation, the engineer discovers that the connection pool between the payment service and the database is exhausted.

A downstream service that normally responds in 10ms started responding in 5 seconds due to a slow query, and the payment service's threads are all waiting for responses.

The connection pool filled up, new requests cannot get a connection, and the entire service is effectively dead while every individual component reports healthy.

This failure mode was completely predictable.

But no one tested for it because traditional testing only validates that things work when everything is working. It does not validate what happens when one thing is slow, one thing is unreachable, or one thing returns unexpected errors.

Chaos engineering fills this gap. It is the practice of deliberately injecting failures into a system to discover weaknesses before they cause real outages.

Instead of waiting for 3 AM to find out that your connection pool does not handle slow downstream services, you simulate a slow downstream service at 2 PM on a Tuesday, observe the behavior, and fix the vulnerability before it becomes an incident.

This guide covers the methodology, the experiments you should start with, the tools available, and how chaos engineering shows up in system design interviews.

What Chaos Engineering Is (and Is Not)

Chaos engineering is not randomly breaking things to see what happens. It is a disciplined, scientific approach to building confidence in your system's resilience.

The discipline was pioneered by Netflix in 2010 when they built Chaos Monkey, a tool that randomly terminated production server instances during business hours.

The reasoning was simple: if Netflix's streaming service could survive random server failures, it could survive the real failures that inevitably happen in cloud environments.

The key distinction is between chaos engineering and breaking things.

Breaking things is uncontrolled destruction with no hypothesis and no measurement.

Chaos engineering is a controlled experiment with a specific hypothesis ("our system should survive the loss of one database replica"), a controlled injection (terminate one replica), careful observation (monitor error rate, latency, and user impact), and a concrete outcome (either the hypothesis holds, or you found a weakness to fix).

For understanding the resilience patterns that chaos engineering validates, A Beginner's Guide to Retry, Circuit Breaker, and Timeout Patterns covers the patterns you are testing when you run chaos experiments.

The Four-Step Experiment Loop

Every chaos experiment follows the same four-step process. This is the scientific method applied to system reliability.

Step 1: Define Steady State

Before you can test what happens during a failure, you need to define what "normal" looks like.

Steady state is measured by the metrics that indicate your system is healthy: P99 latency under 200ms, error rate under 0.1%, throughput at 5,000 requests per second, all health checks passing.

Document these baseline metrics before running any experiment.

If you do not know what normal looks like, you cannot detect when something is abnormal.

Step 2: Form a Hypothesis

A chaos experiment starts with a hypothesis about what should happen during a failure.

Good hypotheses are specific and testable.

"When we terminate one of three application servers, the load balancer should route traffic to the remaining two. P99 latency should increase by no more than 50%, and error rate should stay below 1%."

"When we inject 500ms of network latency between the API service and the database, the circuit breaker should open within 10 seconds and the API should serve cached responses. User-facing error rate should remain below 0.5%."

"When we kill the Redis cache, the system should fall back to database reads. Latency should increase but the system should remain functional. No data should be lost."

A bad hypothesis is vague: "The system should handle failures gracefully."

What does "gracefully" mean?

What failure?

What metrics define success?

Without specifics, you cannot determine whether the experiment passed or failed.

Step 3: Inject Failure

Run the experiment by introducing the specific failure described in your hypothesis. This is where chaos engineering tools come in (covered in the tools section below).

The critical safety rules during injection: limit the blast radius (affect one server, not all servers), have an abort mechanism (a kill switch that stops the experiment immediately), inform the team (everyone on-call should know an experiment is running), and start in staging before moving to production.

Step 4: Observe and Learn

Compare the system's actual behavior during the experiment to your hypothesis.

Three outcomes are possible.

• Hypothesis confirmed: The system behaved as expected. You have increased confidence in your resilience. Document the result and move on to the next experiment.
• Hypothesis falsified: The system did not behave as expected. You found a weakness. This is the most valuable outcome. Fix the weakness, then re-run the experiment to verify the fix.
• Unexpected behavior: Something happened that you did not predict at all. The payment service crashed, but the monitoring system also went down because it depended on the same database. This is the category of discovery that only chaos engineering can provide, because you could not have written a test for a failure mode you did not know existed.

Five Experiments Every Team Should Start With

These five experiments cover the most common failure modes in distributed systems.

Start with experiment 1 and progress in order.

Experiment 1: Kill a Service Instance

What you test: Does the load balancer correctly route traffic away from a dead instance? Does the system maintain availability with reduced capacity?

How to run it: Terminate one application server instance. Monitor traffic distribution, error rate, and latency for the next 5 minutes.

What you expect: Traffic redistributes to healthy instances within seconds. Error rate stays below 1%. Latency increases proportionally to the reduced capacity.

Common failure found: The load balancer's health check interval is too long (30 seconds), causing requests to be routed to the dead instance for up to 30 seconds after it crashes. Fix: reduce health check interval to 5 seconds with a 2-check failure threshold.

Experiment 2: Inject Network Latency

What you test: Do timeouts, circuit breakers, and retry policies work correctly when a downstream service becomes slow?

How to run it: Add 500ms to 2 seconds of latency to all traffic between two services. Monitor the calling service's behavior.

What you expect: The circuit breaker opens after detecting sustained latency. The calling service returns degraded responses (cached data, partial results) instead of timing out.

Common failure found: No circuit breaker is configured. The calling service waits for the full timeout (often 30 seconds) on every request, consuming thread pool resources and eventually becoming unresponsive itself. This is how a slow service causes a cascading failure. For understanding this cascading failure pattern in depth, System Design Interview: How to Prevent Cascading Failures explains the mechanics.

Experiment 3: Fill the Disk

What you test: Does the application handle disk exhaustion gracefully? Do alerts fire before the disk is completely full?

How to run it: Write large files to fill the disk to 95% capacity on one server. Monitor application behavior and alerting.

What you expect: Monitoring alerts fire at 80% disk usage. The application logs an error when writes fail but does not crash. Other services are not affected.

Common failure found: The application crashes with an unhandled exception when a log file write fails. No alert exists for disk usage, so the team only learns about it when the application goes down. Fix: handle write errors gracefully, add disk usage alerting at 80%, and implement log rotation.

Experiment 4: DNS Failure

What you test: What happens when DNS resolution fails for a downstream service? Does the application use cached DNS entries? Does it retry with backoff?

How to run it: Block DNS resolution for one downstream service for 30 seconds. Monitor the calling service's behavior.

What you expect: The application uses cached DNS entries for the duration of the failure. If no cached entries exist, it retries with exponential backoff and returns a degraded response after exhausting retries.

Common failure found: The application has no DNS caching and makes a fresh DNS lookup for every request. When DNS fails, every request fails immediately. Fix: configure local DNS caching with a reasonable TTL and implement retry logic for DNS resolution failures.

Experiment 5: Kill the Cache

What you test: What happens when Redis or Memcached goes down? Does the application fall back to database reads? Does the database handle the sudden load increase?

How to run it: Terminate the Redis instance. Monitor application latency, database load, and error rate.

What you expect: The application detects the cache failure and falls back to direct database reads. Latency increases but the system remains functional. No data is lost.

Common failure found: The application treats cache failures as hard errors and returns 500 to all users, even though the data exists in the database. Or the database is overwhelmed by the sudden traffic that the cache was absorbing, leading to a cascading failure. For understanding the broader set of reliability techniques to prevent these cascading issues, 8 Techniques for Building Reliable Distributed Systems covers the resilience patterns.