What to Actually Alert On in a Zero Trust API Setup

A single token rejection because a session expired is expected behavior, not an incident. A sudden spike in rejections from one client, or rejections from a client that has never had one before, is a different signal entirely. Build your alerting around the second category (rate of change, first occurrence, unusual concentration) rather than raw counts of denials, which will always be high in a healthy system doing its job.

Write down, per signal, what "normal" looks like on a typical day, so an anomaly detector or a human reviewing a dashboard has something to compare against.

Imagine a client that normally sees a handful of expired-token rejections a day because its refresh logic occasionally lags: that's baseline noise. The same client suddenly generating rejections on every single call is the anomaly, and the difference between the two is the rate of change against its own history, not an absolute number that applies to every client the same way.

Expiring mTLS certificates and rotating API keys cause a specific, predictable failure mode: everything works fine until a specific date, then a service starts failing every request. This deserves advance warning, days ahead, not a same-day alert when it's already too late to fix calmly. Put expiry warnings on their own schedule-based check rather than mixing them into your general anomaly alerting, since they need a different kind of response (renew before the deadline) than a security anomaly does (investigate now).

A common mistake is treating certificate expiry as a one-time setup task instead of a recurring operational one: the team that provisioned the cert two years ago has often moved on to other work by the time it expires, and the alert fires against an on-call rotation that has no context for what the certificate is even for. Document what each certificate protects, not just when it expires, so whoever gets the alert can act on it without archaeology.

An alert with no defined first action just interrupts someone at 2 a.m. so they can look at it and go back to sleep. For every alert in your matrix, write one sentence describing what the on-call engineer actually does when it fires: check this dashboard, run this query, page this other team. If you can't write that sentence, the alert isn't ready to page anyone; downgrade it to a dashboard metric until it is.

Do this review quarterly. Alerts that made sense at your old scale often stop making sense as traffic patterns and team structure change.

Your error budget, and how it maps to your availability target, is the right lens for a security-observability dashboard: it converts a stream of individual denial events into a single number your team can track against a real commitment. The gap between a 99.9% and a 99.99% target is the difference between roughly nine hours and roughly fifty minutes of allowed downtime a year¹, and that budget should be visible on the same dashboard as your security denial rate, since a misconfigured policy that starts rejecting valid traffic burns the same budget as an infrastructure outage.

After any real security incident, however small, go back to your alerting matrix and ask whether the signal that would have caught it earlier existed, and if it did, why it didn't page anyone. This is a better source of new alerting rules than brainstorming in the abstract, because it's grounded in something that actually happened rather than something that theoretically could.

Keep the review short and specific: one incident, one question (would an alert have caught this sooner, and why didn't the existing ones), one concrete change to the matrix. A long, general postmortem meeting tends to produce vague action items that never turn into an actual new alert; a narrow one produces a rule you can point to next quarter.

Each row in your alerting matrix should record:

• The signal itself and what normal looks like on a typical day, so anomalies have a baseline to compare against.
• Whether the signal pages someone, appears on a dashboard, or is only logged for later review.
• One sentence describing the first action the on-call engineer takes when it fires, or the alert gets downgraded.
• For certificate and credential expiry, a schedule-based warning days ahead instead of a same-day alert.