A Capacity Planning Runbook for Teams Tired of Fire Drills

Every service that can fail under load needs an explicit ceiling: a level of CPU, memory, connection pool usage, or queue depth past which you consider it in trouble. Say you decide no stateful service should sustain more than 70% memory utilization for longer than 15 minutes. That number becomes the thing people check, not "does it feel slow." Pick the number with the team that owns the service, write it down next to the service's runbook, and revisit it after any architecture change. A target picked in a meeting six months ago and never revisited is worse than no target, because it gives false confidence.

CPU and memory are lagging indicators: by the time they are pinned, you are already in the incident. Queue depth, connection pool wait time, and the growth rate of your busiest table or index are leading indicators, because they tell you where you are heading before you arrive. For a queue-backed service, a queue that is growing faster than it drains for more than a few minutes is a better signal than any percentage. Build your alerting around the trend, not just the threshold. Say a service is climbing toward its ceiling a little more each day; that trend deserves a ticket even while the service is still comfortably under the line, because by the time it crosses the line the fix usually takes longer than the warning gave you time for. A dashboard that only turns red at the ceiling is a dashboard that tells you about the incident after it has already started.

Stateless compute behind an autoscaler can absorb a spike in minutes. A database's storage volume, a message broker's partition count, a third-party API's rate limit, and a cloud provider's quota for a given instance family cannot. Make a short list, per service, of which dependencies fall into the slow category, and treat a request to raise one of those limits as something you file weeks ahead of a known event, not the week before. If you cannot tell how long a given increase takes, ask the vendor or your cloud account team directly instead of assuming it is instant. Quota increases in particular are an easy thing to forget, because they sit outside your own codebase and rarely show up in a normal deployment checklist; the team that remembers them is usually the team that got burned by one once and wrote it down afterward so nobody had to learn it twice.

Before any event you expect to change traffic, work through the arithmetic on paper first. Say your current peak is a known number of requests per second and marketing is planning a push that historically multiplies traffic for a week. Multiply your peak by that factor, check it against every layer the request touches (load balancer, application tier, cache, database, any rate-limited third party), and find the layer that breaks first. That is your bottleneck, and it is usually not the layer everyone assumed; teams tend to assume the database will be the constraint and are often surprised to find a third-party API's rate limit or a cache eviction policy gets there first. Fix that one layer, then redo the math, rather than adding capacity everywhere evenly, since evenly-distributed extra capacity is usually a more expensive way to solve a problem that lives in one specific place.

Autoscaling is right for capacity you can add and remove in minutes with no state to carry: web and application tiers, most background workers. Pre-provisioning is right for anything with a cold-start cost, a contractual minimum, or state that cannot simply appear on demand: database read replicas, cache cluster size, reserved capacity you have committed to for cost reasons. Write this decision down per layer once, so the next person planning a launch is not re-litigating whether the database read replica should "just autoscale." It cannot, and pretending otherwise is how launches go wrong late at night when the on-call engineer discovers, mid-incident, that the thing they assumed would scale itself needed a change request filed days earlier.

Sort each layer into one of these two groups:

• Autoscale web and application tiers and most background workers, since capacity can be added and removed in minutes with no state to carry.
• Pre-provision database read replicas and cache cluster size, where state or cold-start cost means capacity cannot simply appear on demand.
• Pre-provision anything under a contractual minimum, such as reserved capacity you have committed to for cost reasons.
• Start requests to raise slow-scaling limits, like storage volume, partition count, API rate limits and instance quotas, well ahead of the launch.

A capacity plan that lives only in one engineer's memory disappears the day that engineer is on vacation or has moved to a different team. Keep a short, living document per service: its headroom target, its slow-to-expand dependencies, and the layer that broke first the last time someone ran the arithmetic. Update it after every launch, whether or not anything went wrong, since a launch that went smoothly still tells you something useful about where the next bottleneck is likely to be.