AI Feature Flags Cheatsheet: What to Gate, Rollback, and Monitor

For: A product engineer at a 30–150 person B2B SaaS company who just shipped an AI feature to 100% of users, got a wave of support tickets about wrong or weird outputs, and now needs to roll it back — but realizes their feature flag setup was designed for UI changes, not probabilistic model behavior, so they have no clean way to segment affected users, pin a model version, or define what 'bad output' even means as a rollback trigger

Standard feature flags gate who sees a feature. AI features need a second layer that gates how the model behaves — version, prompt template, embedding schema, retrieval index, and output quality thresholds. If your rollback story is a single boolean, you can only ship 100% on or 100% off. This cheatsheet covers the flags to add, the rollback triggers to define, and the signals to monitor.

The two-layer flag model

Every AI feature should sit behind two independent flag layers. If you only have one, the same code path can produce wildly different outputs the moment a model or prompt changes underneath you.

The behavior layer is what lets you roll back a bad prompt without turning the feature off entirely.

What to gate

Always gate these

• Model provider + version — pin explicit versions (gpt-4o-2024-08-06, not gpt-4o). Providers silently update aliases.
• Prompt template version — treat prompts like migrations. Store hashes.
• Temperature and sampling params — a prod change from 0.2 → 0.7 is a behavior change.
• Retrieval config — embedding model, index version, top-k, reranker on/off.
• System prompt / tool definitions — separate flag from user-facing prompt.
• Guardrail chain — PII redaction, refusal classifier, output validator.
• Max tokens / cost ceiling per request — kill switch for runaway generations.

Gate per-tenant when

• Tenants have different data sensitivity (healthcare vs. marketing).
• Contract SLAs differ on accuracy or latency.
• You're running a paid preview or design partner cohort.
• A tenant has opted out of training-data usage — different provider routing.

Don't bother gating

• Minor logging changes, prompt whitespace fixes, or fully deterministic post-processing.
• Retry logic changes (test in staging; flagging adds noise).

Rollout patterns for AI features

The mistake most teams make: jumping straight from staging to a 100% eligibility rollout with no canary on the behavior layer. Ship the feature to 100% of eligible users, then ramp the new model version from 1% → 100% behind a second flag.

Rollback triggers — what counts as "bad"

You cannot roll back on "users complained." Define numeric triggers before shipping. Each trigger needs a threshold, a window, and an action.

Tradeoff: automatic rollback on thumbs-down feels safe but is easy to trigger with a single angry power user. Require a minimum sample size and a comparison against a rolling baseline, not an absolute number.

What to monitor per rollout stage

Pre-launch (shadow)

• Output diff rate vs. control (semantic similarity, not string equality)
• Token usage distribution — spot prompt regressions
• Tool-call success rate for agent features
• Eval score on a frozen golden set of 200–500 examples

Canary (1–25%)

• All of the above, plus user feedback signals
• Segmented by tenant tier, locale, and input length — averages hide localized failures
• Time-to-first-token and full-completion latency separately

Full rollout

• Daily eval on golden set (catches provider-side drift)
• Cost per active user, week-over-week
• Feature adoption vs. abandonment mid-interaction (users canceling generations)

Minimum flag schema

{
  "feature": "ai_summary",
  "eligibility": { "tenants": ["..."], "plans": ["pro","enterprise"] },
  "behavior": {
    "model": "claude-3-5-sonnet-20241022",
    "prompt_version": "summary_v7",
    "temperature": 0.2,
    "max_tokens": 800,
    "retriever": { "index": "docs_v3", "embed_model": "text-embedding-3-large", "top_k": 6 },
    "guardrails": ["pii_redact_v2","json_validator_v1"]
  },
  "rollout": { "stage": "canary", "percent": 5, "fallback": "summary_v6" },
  "triggers": { "thumbs_down_ratio": 0.15, "validator_fail_pct": 1.0 }
}

Two details that matter: an explicit fallback configuration (not just "off"), and the trigger thresholds shipped with the flag itself so ops teams don't have to hunt through dashboards.

Common mistakes

• Using LaunchDarkly / Unleash / Flagsmith as-is. They gate booleans well but weren't designed for versioned behavior payloads. Either extend with a config service or store behavior configs separately keyed by flag variant.
• No warm fallback. Rolling back a prompt means the previous version must still be loaded, tested, and callable. Cold-restore is not rollback.
• Flag sprawl. One flag per feature, with a config payload. Not 14 flags per feature.
• Eval-in-prod only. A golden set of 200–500 examples with a scorecard beats waiting for thumbs-downs.
• Ignoring provider drift. Even pinned model versions get deprecated. Track deprecation dates as first-class flag metadata.

Teams building on top of LLM APIs — especially in regulated verticals like healthcare or lending — usually end up building this two-layer setup after their first production incident. Building it before the incident is cheaper. More on how we approach production AI systems on the AI Studio page.

Frequently Asked Questions

Can I use LaunchDarkly or Unleash for AI feature flags?

Yes for the eligibility layer — they're excellent at that. For the behavior layer (model version, prompt version, retriever config), you'll want either multivariate flags carrying JSON payloads or a separate config service keyed by flag variant. The out-of-the-box boolean model isn't enough on its own.

What's the difference between a canary release for ML and a normal canary?

A normal canary looks at error rates and latency. An ML canary also needs output-quality signals — thumbs-down ratio, validator failure rate, refusal rate, and comparison to a golden eval set — because the code path can be healthy while the outputs are wrong. You also segment by input type, not just user cohort, because failures cluster around specific input shapes.

How do I roll back an AI feature without turning it off completely?

Keep the previous behavior config (model + prompt + retriever versions) warm and callable, and make your flag variant point to it as a fallback. Rollback then becomes a variant switch, not a feature disable. This is why the behavior layer must be independent from the eligibility layer.

What should trigger an automatic rollback vs. a human review?

Automate on hard signals with clear baselines: validator failures, schema violations, latency breaches, and refusal-rate spikes above a minimum sample size. Route soft signals — thumbs-downs, support tickets, subjective quality drops — to human review. Auto-rollback on subjective signals produces too many false positives.

How large should a golden eval set be?

Start with 200–500 curated examples covering your top intents, edge cases, and known past failures. Grow it every time an incident produces a new failure mode — regressions caught by a golden set are cheaper than regressions caught in production. Rerun it on every behavior-layer change before promoting beyond canary.