Handling GDPR Erasure Requests in a Streaming Pipeline

Not by deleting it from the topic alone. Once a message has been read by every consumer that was going to read it, deleting it from the broker doesn't touch the copies those consumers already wrote into their own stores. An erasure request against streamed data has to reach every downstream system that materialized a copy, not just the original topic.

The practical answer is a registry: track which consumers persist data from which topics, so an erasure request can be routed to every place a copy might exist instead of relying on someone to remember all of them during an actual request.

A compacted topic keeping the current value per key indefinitely is exactly the pattern that makes an erasure request straightforward for that data: writing a tombstone (a null value) for the key removes the current state during the next compaction cycle. This is one case where the streaming pattern actually helps rather than complicates things, as long as your consumers are built to treat a tombstone as a real deletion rather than something to ignore.

Data minimization is easier to satisfy going in than to fix after the fact. Prefer passing a reference key (an account ID) over the full record in the event itself, and let consumers that genuinely need the personal details look them up from a system of record that has its own access controls and its own erasure path.

This also shrinks your erasure surface directly: a stream carrying IDs instead of names, emails, and addresses has far fewer places personal data can end up copied into a downstream store you'd otherwise have to track down.

Cross-border transfer restrictions are a well-established part of GDPR, and a multi-region streaming setup can move data across borders in ways that aren't obvious from the application code, particularly if replication or disaster recovery quietly copies a topic to a broker in another region. Map where your brokers and any cross-region replication actually run, and confirm with your counsel whether your current transfer mechanism covers that path specifically, since the right mechanism depends on where data moves and what safeguard applies to that specific route.

Vanta and Drata can track that your data processing agreements are current and that privacy policy acknowledgments happened on schedule, which is useful evidence to have organized. Neither one can trace which of your consumers persisted a specific customer's data or write your erasure logic for you.

The registry of consumers and their downstream stores mentioned earlier is pipeline-specific engineering work; a compliance platform's value shows up after that groundwork exists, in keeping the paperwork around it current.

Keep a record that a request came in, which systems it was routed to based on your consumer registry, and confirmation from each one that the data was actually removed, without logging the personal data itself in that record. This gives you something to show if a regulator or a customer asks whether a request was actually completed, without creating a new place personal data sits longer than it needs to.

Set a reasonable internal deadline for closing out a request across every downstream system, and treat a system that hasn't confirmed by that deadline as an open item to chase, not something to assume completed silently.

When an erasure request arrives, work through these steps:

1. Look up the subject's identifier in your consumer registry to see which consumers persist data from which topics.
2. Route the request to every downstream system that holds a copy, not just the original topic.
3. For a compacted topic, write a tombstone for the key so the next compaction cycle removes the current state.
4. Get confirmation from each system that the data was actually removed.
5. Log that the request arrived, where it was routed, and each confirmation, without recording the personal data itself.