Why Your Log Bill Grows Faster Than Your Traffic

A service logging at debug level in production, because nobody changed the default after the initial setup, can easily produce ten times the volume of the same service logging only what an operator would actually need during an incident. Audit your highest volume log sources first, not your newest ones: the biggest single fix is usually one or two chatty services logging far more than anyone reads, not a broad problem across the whole fleet.

Reducing log detail per line saves a little. Sampling, keeping every error and a representative slice of successful requests instead of logging every single one, changes the actual cost curve, because the volume driver in most systems is the high frequency, low information successful request, not the rare, information dense error. Set your sample rate so every error is always kept in full and successful requests are sampled at whatever rate your budget allows.

Most log queries happen against the last few days; queries against data older than a month are rare and usually driven by a specific investigation, not routine monitoring. A three tier setup, hot storage for the last week where every field is instantly searchable, warm for the last quarter at lower cost and slower query speed, cold archival storage beyond that, matches spend to actual access patterns instead of paying hot storage prices for data nobody's queried in months.

Free text log lines are cheap to write and expensive to search: finding every request that hit a specific error requires a fuzzy text search across everything. Structured, field based logging costs a little more discipline upfront but turns that same search into an exact filter, which matters directly during an incident when a 99.95% uptime target leaves roughly 4.38 hours a year of downtime budget to work with across every incident combined¹, and every minute spent searching unstructured text is a minute of that budget spent on the search instead of the fix.

Logging everything indefinitely feels safer than deciding what to drop, but it's a decision by default, not a real strategy, and it's usually the single biggest line item on the bill. The honest question isn't whether a piece of data might someday be useful, almost anything might be, it's how often you've actually gone looking for data that old, and at what level of detail. Most teams have never once queried logs from six months ago at full per-request granularity.

Keep every error, every security relevant event, and every request touching payment or auth flows in full, regardless of volume. Sample routine successful requests aggressively once volume is high enough that full retention is clearly the biggest cost driver. Review the rule quarterly rather than setting it once, since what counts as routine shifts as the product changes, and a rule set a year ago may be sampling away exactly the traffic that's become worth watching closely now.

Use these rules to decide what to keep whole:

• Keep every error in full regardless of volume, since errors are rare and carry the most information.
• Keep every security relevant event, and every request touching payment or auth flows, in full.
• Sample routine successful requests aggressively once full retention is clearly the biggest cost driver.
• Review the rule quarterly rather than setting it once, since what counts as routine shifts as the product changes.

Say a single payments service is logging at debug level and accounts for sixty percent of total log volume, while every other service combined accounts for the rest. Dropping that one service to info level in production, keeping debug available on demand for local debugging only, can cut total volume by more than half without touching retention or sampling anywhere else. Before building a broader sampling and tiering strategy, check whether your bill actually has a concentration problem like this one: it's a much faster fix than redesigning the whole pipeline.