LLMs, GenAI, RAG, agents, evaluation · insight-1623

Build a feature-level kill switch and a degraded mode before launch

Situation
A customer-facing LLM feature is about to ship and the incident plan is "redeploy".
Action
Provide runtime flags the on-call can flip without a deploy: whole feature off, tools off, one retrieval source off, switch to a fallback response, model or human handoff, globally or per tenant. Log who flipped what, rehearse quarterly and page on spend, guardrail-hit and tool-volume thresholds.
Reason
Time to contain is a design property. NIST's generative-AI profile suggests protocols so that systems can be deactivated when necessary.
Boundary
Any generative feature with real users, tools or spend exposure.

Symptom or risk

When harmful output, a leak or runaway spend starts at night, stopping it needs a code change, a review and a deploy while harm continues.

Internal experiments behind a single owner's account.

The fallback path must be built and maintained, and the degraded product is worse.

A switch that hides the UI but leaves the API or batch path live, and a flag service whose default on outage is "on".

NIST AI 600-1 action GV-1.7-001 suggests putting protocols in place so that generative-AI systems can be deactivated when necessary.
Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile