Capacity Planning

4. Whole Machines, in a Few Sizes

Arithmetic gives you a number of machines with a decimal point. Vendors sell whole machines, in a small set of sizes, and the gap between those two facts is where capacity plans lose accuracy.

• You cannot buy 3.7 machines. You buy four, and four is the honest number. Across a small fleet this rounding is a large fraction of the total.
• Instance sizes are coarse. If the next size up is double, then needing 10 percent more capacity than one size can mean paying for 100 percent more.
• Workloads have more than one dimension. A service that exhausts memory while using a fifth of its CPU does not pack with one that does the reverse. Capacity is not a single number; it is whichever dimension runs out first, and that dimension differs per service.
• Stranded capacity is real. A machine with plenty of CPU left and no memory left is full. Fleet utilization figures that average across dimensions hide this and routinely overstate how much room remains.

So plan capacity in the dimension that binds, name it out loud, and check the other dimensions for stranding. A fleet reported as 60 percent utilized may have no usable space at all.

5. What Autoscaling Does Not Fix

Automatic scaling is often treated as a reason not to plan capacity. It changes the problem rather than removing it, and it has four limits worth knowing before relying on it.

• It reacts more slowly than traffic arrives. Detecting load, starting an instance, warming it and passing health checks takes from tens of seconds to several minutes. A spike that arrives in ten seconds is served entirely by the capacity you already had.
• Something must survive the gap. Whatever the scale-up delay is, your existing fleet handles the whole spike for that long. That sets a floor on steady-state capacity that autoscaling cannot lower.
• It scales what you told it to. Adding application instances when the database is the constraint makes the problem worse, because each new instance opens its own connections and adds load to the thing that was already saturated.
• Scaling down has its own risk. Aggressive scale-down leaves you with the wrong floor when traffic returns, and scale-down during a partial failure removes capacity exactly when the survivors need it.

Autoscaling is genuinely good at following a daily curve and at absorbing moderate, gradual growth. It is poor at absorbing sharp spikes and it does nothing for a bottleneck that is not the thing being scaled.

6. The Senior Decision

A capacity answer is not a number. It is four statements, and leaving any of them out makes the number impossible to check.

1. Which peak. "The busiest minute of the busiest day we expect this quarter."
2. What headroom, and why. "Sized to 65 percent, so that losing one of three zones keeps us under capacity."
3. What horizon, and at what growth. "Six months out at 10 percent monthly, which is 1.8 times current load."
4. Which dimension binds. "Memory, at roughly 40 gigabytes per instance; CPU is at a third."

Stated that way, every assumption is visible and arguable, which is exactly what you want. A reviewer can disagree with the growth rate without rejecting the whole plan, and the plan can be revisited by changing one input.

What the interviewer is scoring: "how many servers do you need?" is a question about assumptions, not arithmetic. The weak answer produces a number. The strong answer produces a number with its peak, its headroom, its horizon and its binding resource attached, and it says which assumption the answer is most sensitive to. Two follow-ups are near certain. "What happens if you lose an availability zone?" is checking whether your headroom was chosen or inherited. "Why not just autoscale?" is checking whether you know that scaling takes longer than a spike does, and that the existing fleet carries everything until it finishes.