Rotating API Keys With Zero Downtime: A Step-by-Step Playbook

Rotations fail for predictable reasons: the old key is revoked before all consumers have the new one, a service caches the secret at startup and never reloads it, or a forgotten script uses the key from a laptop or a cron job. Each of these is a coordination problem, not a cryptography problem.

Outages also cost real budget. A 99.9 percent availability target allows about 8.76 hours of downtime a year¹, so a rotation that breaks production for even a quarter of an hour uses a visible slice of it. The fix is a design that can't fail closed: keep two valid keys during the transition and prove the old one is idle before you kill it.

Follow this sequence:

1. Inventory consumers. List every service, job, pipeline and person that uses the key. Search the repositories and your secrets store.
2. Create the new key at the provider without disabling the old one. Confirm the provider allows two active keys at once. If it doesn't, plan a brief switch window and pick a quiet time.
3. Store the new key in your secrets manager as a new version, not by overwriting the old one in place.
4. Roll it out in stages. Update a low-risk consumer first, then verify requests succeed with the new key. Continue service by service.
5. Watch for the old key. Check provider logs for last-used times on the old key.
6. Revoke the old key only after it has shown no traffic for a period you set, such as a full business cycle.
7. Record the date, the owner and the next rotation due.

Secrets tools such as Doppler, HashiCorp Vault or AWS Secrets Manager are options for storing versioned secrets and getting a new value to your services, which can make step four safer. Confirm in a demo how each fits your stack.

If you issue API keys, build rotation into the product so customers never face a cliff:

• Allow multiple active keys per account. Customers create key two, deploy it, then delete key one.
• Give each key an identifier and a label, and show last-used time, so customers can see stragglers.
• Support expiry with a grace period, so an expiring key still works briefly and returns a clear warning header or message.
• Notify before expiry by email and in the dashboard.
• Store only hashes of keys on your side and show the full value once at creation.
• Offer an emergency revoke that takes effect immediately for a leaked key.

The common mistake is a single key per account, where rotation means a hard cutover and downtime for every integration.

Design services so a new secret doesn't require a redeploy:

1. Read secrets at runtime, from the secrets manager or a mounted file, instead of baking them into images or build-time variables.
2. Reload on change or on failure. If a request fails authentication, fetch the latest secret once and retry before returning an error.
3. Accept two values on the receiving side for shared secrets, such as webhook signing keys, by verifying against both the current and the previous key during a transition.
4. Use short-lived credentials where the platform allows it, so rotation happens automatically and nobody handles a long-lived key.
5. Keep secrets out of logs and error messages, since a rotated key that leaked into logs stays exposed.

For picking a store, see the comparison of HashiCorp Vault, AWS Secrets Manager and Doppler.

An emergency rotation compresses the same steps, so rehearse it before you need it:

• Revoke first if the exposure is severe, accepting a short outage over continued misuse. If the key can move money or expose customer data, don't wait for a clean overlap.
• Rotate related secrets. A leaked deployment token may expose others in the same environment.
• Check the provider's logs for activity from unfamiliar addresses during the exposure window.
• Remove the secret from history where it appeared, but assume it's compromised anyway, since deleting a commit doesn't undo who saw it.
• Write a short incident note covering how it leaked and what prevents a repeat.

Set a routine schedule too. Many teams rotate on a calendar and after every offboarding of someone with access. For a more careful approach to environment files, see the Doppler vs AWS Secrets Manager comparison for B2B SaaS.