How to Run a Real Security Audit on a Distributed System

Start with an honest inventory, not the architecture diagram from the last fundraising deck. List every service, who owns it, what data crosses its boundary, and whether it is reachable from the internet or only from inside your network. Most distributed systems have at least a few services nobody has looked at since the person who built them left.

For each service, write down what calls it and what it calls. An internal billing service that quietly accepts requests from twelve other services is a bigger governance problem than a single public API, because a compromise anywhere in that chain reaches it.

An auditor will ask how fast you fix a known vulnerability, and "whenever someone gets to it" is not an answer. Federal patch rules are a reasonable baseline to borrow: critical, internet-facing vulnerabilities remediated within 15 days, and anything already being exploited fixed within 14¹.

Write the SLA down, tie it to severity, and track how often you actually meet it. A documented target you miss half the time is still more useful to an auditor, and to you, than no target at all, because it shows you where the process is actually breaking.

Most breaches in distributed systems trace back to a credential, not a zero-day. Pull the list of service accounts and API keys and check three things for each: who or what still uses it, when it last rotated, and whether its permissions are broader than the one job it does.

A service account created for a migration two years ago that still has write access to production is a common finding, and it is an easy one to fix once you actually go looking for it.

Scope matters as much as rotation. A key that's rotated on schedule but still has admin rights over every database in the account gives an attacker the same blast radius as a key that never rotates at all. Walk each credential down to the narrowest permission set that still lets its owning service do its job, and treat any exception to that rule as something that needs a written reason, not a default.

A few patterns show up in almost every distributed system that hasn't been audited recently:

• A stale service inventory, so the audit misses services nobody remembered to list
• Shared credentials used by multiple services, so revoking one means breaking three
• No logging on internal service-to-service calls, so a lateral move is invisible after the fact
• Exceptions granted during an incident that were never revisited once things calmed down

Any one of these turns a routine audit into a much longer project, so check for them first.

A point-in-time audit tells you where you stood on the day someone looked. What actually holds up under governance review is continuous evidence: access reviews that run monthly, vulnerability scans that feed a dashboard, and change logs that timestamp themselves.

Tools built for this, like Vanta and Drata, automate the collection so you are not rebuilding the same spreadsheet every quarter. If your team is small, that automation often pays for itself the first time you avoid a week of manual evidence gathering.

The failure mode isn't usually a missed finding, it's a found one that goes nowhere. An audit that produces a long list of issues with no owner, no deadline, and no follow-up review is worse than not auditing at all, because it creates a false sense that someone is tracking the risk.

Every finding needs three things before the audit is considered closed: a named owner, a date, and a second look to confirm the fix actually shipped. If a finding has sat open for two audit cycles in a row, that's a signal the ownership model, not the finding itself, is the real problem.

If your engineering lead wants a second opinion on how a specific finding maps to risk, Taj, MeetMyCTO's AI CTO, can walk through the tradeoffs against your actual setup in the advisor panel.