What a Real Security Audit of Model Serving Should Cover

A typical audit checks authentication, input validation, and dependency versions. A model-serving audit needs all of that plus three things a CRUD API doesn't have: the model artifact itself, the prompts and completions flowing through it, and the GPU or accelerator layer underneath.

Model weights are a business asset and sometimes a licensed one. If a fine-tuned checkpoint sits in a storage bucket with the same permissions as your marketing assets, that's a gap. Prompts and completions often carry customer data even when the request looks like a generic API call, so your data classification has to extend into the inference layer, not stop at the database.

The GPU layer adds its own risk: a shared inference cluster that isn't tenant-isolated can let one customer's traffic pattern, or a cached activation, bleed into another customer's session.

Before you call the audit done, walk through the endpoint itself:

• Every inference route requires authentication, not just an API key sitting in a query string.
• Rate limits exist, but they're a cost control, not a substitute for authorization checks.
• Traffic between your gateway and the model server runs over an encrypted internal channel, not the open internet.
• Raw prompts and completions are redacted or tokenized before they hit your logging pipeline.
• Inference API keys are scoped per caller and rotated on a schedule, not shared across every internal service.

Inference servers carry the same class of vulnerabilities as any other network service, and they sit closer to your GPUs than a typical app server does. Treat their CVEs with urgency, not a quarterly patch window.

Federal guidance is a reasonable floor to hold yourself to even outside government work: critical, internet-facing vulnerabilities get patched within 15 days, and high-severity ones within 30¹. If your inference gateway is reachable from the internet at all, hold it to the critical timeline, not the high one.

Container base images for your serving stack should be rebuilt on the same cadence, not left running for months because the model still works fine.

Vanta and Drata are built to collect and organize the evidence an auditor asks for: access reviews, patch records, vendor questionnaires. They're useful for exactly that, and they cut down the manual screenshot-and-spreadsheet work a SOC 2 or ISO audit usually creates.

What they don't do is test your inference endpoint for the model-serving-specific issues above. A clean compliance dashboard doesn't tell you whether your model weights are readable by every engineer with storage access, or whether a prompt injection can make your model call a tool it shouldn't. Use compliance automation for the paperwork side, and a separate technical review for the rest. If you're deciding between platforms, see Vanta vs. Drata vs. Secureframe.

A first audit of this kind tends to miss the same handful of things:

• Treating a third-party model API as out of scope because you don't host it, when you're still responsible for what you send it.
• Forgetting that a fine-tuned checkpoint is a separate asset from the base model, with its own access list.
• Auditing the production model server but skipping the staging one, which usually has weaker controls and the same data.
• Assuming rate limiting stops abuse, when it only slows it down.

Fix these before you fix the smaller stuff; they're the ones that turn into incidents.