The Feature Flags Nobody Remembers Turning On

A flag created to test a new reranker should be tagged not just with what it does, but with what has to be true for it to be safe to remove: the experiment concluded, a winner chosen, the losing path deleted. Without that condition written down at creation time, a flag's purpose fades from memory long before the flag itself gets removed, and eventually nobody feels confident deleting something they don't fully understand anymore.

A flag meant to gate a temporary experiment sometimes becomes the de facto way a permanent decision gets made, an embedding model choice or a retrieval parameter that's actually fixed in practice but still lives behind a flag because removing the flag means touching code nobody wants to touch. If a flag's value hasn't changed in months and isn't expected to, it's not a flag anymore, it's configuration wearing a flag's clothing, and it should be simplified into an actual config value or removed.

A flag controlling which embedding model is active and a flag controlling which reranker is active can combine into states nobody tested: the new embedding model with the old reranker, or the reverse, if both flags can be toggled independently. Map out which combinations of your active flags are actually valid, and either prevent invalid combinations in code or explicitly test them, rather than assuming flags that were each tested individually are safe together.

For example, suppose one flag switches between an old and a new embedding model, and another switches between an old and a new reranker. Each was tested on its own, but the new embedding model paired with the old reranker was never run together, and that pairing appears whenever the two flags are rolled out at different speeds. A simple fix is to write down the valid combinations, reject the rest in code, and add a test for each allowed pair. That takes a small amount of effort and removes a whole class of surprises that no single flag review would ever catch.

When an experiment concludes and a variant loses, removing the flag without removing the losing code path leaves dead logic that still has to be read, understood, and accounted for by anyone touching that part of the pipeline later. Treat flag removal as removing the flag and the losing branch together, in the same change, so the codebase actually gets simpler after an experiment ends instead of just gaining one more piece of history nobody prunes.

Set a recurring review, monthly or quarterly depending on how fast your team adds flags, where someone goes through the full active list and asks, for each one, whether its removal condition has been met. Flags that survive several reviews without their condition being met are worth a direct question to whoever owns them: is this actually still an open experiment, or has it just been forgotten.

• Does every flag have a written removal condition from the day it's created, not just a description of what it does?
• Are any flags actually functioning as permanent configuration rather than a temporary experiment?
• Have the valid and invalid combinations of interacting flags been mapped and tested?
• Does removing a flag also remove the losing code path, in the same change?
• Is there a recurring, scheduled review of the full active flag list?

A flag list reviewed by a team collectively tends to produce the same outcome every time: everyone assumes someone else knows why a given flag is still there, and nobody removes it. Assign each flag an individual owner at creation time, the person who can actually answer whether its removal condition has been met, so the recurring review has someone specific to ask instead of a group shrug. When that person leaves the team or changes roles, reassign their flags explicitly rather than letting ownership quietly lapse along with them.