Kill Switch

A kill switch is a control that lets an operator turn off a function in a running system, typically within seconds, and without deploying code. It is the operational kind of feature flag: Pete Hodgson’s guide to feature toggles notes that such operational toggles must be reconfigurable extremely quickly, since needing a new release to flip one defeats its purpose. It can also be a manually operated circuit breaker, a protective switch left under a person’s control. Having a flag called “kill switch” does not make it one; whether it works in an incident depends on a few conditions.

Check it where the effect happens

Suppose an agent can call a refund tool, and an operator turns the refund switch off at 300 seconds. If the switch is read once, when a run is admitted, every run already in progress will still reach its refund step. If the tool executor reads the switch just before it executes the refund, those runs stop. The simulation below sends about two runs per second for ten minutes. Each run lasts 45 seconds, the refund lands at a random point inside the run, and three replicas poll the switch every 10 seconds at different offsets. The traffic and timings are invented. It ran on Python 3.12.14 on macOS.

import random

POLL = 10          # each replica re-reads the flag store every POLL seconds
PHASES = (0, 3, 7) # three replicas poll at different offsets
FLIP = 300         # the operator turns the refund tool off at t=300 s
RUN_LEN = 45       # an agent run lasts 45 s; its refund call lands somewhere inside it
r = random.Random(5)
runs = []          # (start, refund_time, replica)
t = 0.0
while t < 600:
    t += r.expovariate(2.0)             # about 2 runs per second
    runs.append((t, t + r.uniform(5, RUN_LEN - 5), r.randrange(3)))

def flag_on(now, replica):
    ph = PHASES[replica]
    seen = ((now - ph) // POLL) * POLL + ph      # last poll this replica made
    return seen < FLIP                            # the flip is visible once a poll lands after it

# Variant 1: the flag is read once, when the run is admitted.
v1 = [f for s, f, k in runs if flag_on(s, k) and f >= FLIP]
# Variant 2: the flag is read by the tool executor, just before the refund runs.
v2 = [f for s, f, k in runs if flag_on(f, k) and f >= FLIP]
inflight = sum(1 for s, f, k in runs if s < FLIP <= f)
print(f"runs started: {len(runs)}; runs in flight at t={FLIP} s with a refund still to come: {inflight}")
print(f"refunds executed after the flip, flag checked at admission: {len(v1)} (last at t={max(v1):.0f} s)")
print(f"refunds executed after the flip, flag checked at execution: {len(v2)} (last at t={max(v2):.0f} s)")

print()
print("flag store unreachable from t=200 s; bad release from t=240 s makes half of later refunds wrong")
good = bad = 0
for s, f, k in runs:
    if f >= 200:
        if f >= 240 and r.random() < 0.5: bad += 1
        else: good += 1
print(f"refund calls after t=200 s: {good + bad} ({bad} wrong, {good} correct)")
print(f"{'on read error':<34}{'wrong executed':>16}{'correct blocked':>17}")
print(f"{'fail open (treat as enabled)':<34}{bad:>16}{0:>17}")
print(f"{'fail closed (treat as disabled)':<34}{0:>16}{good:>17}")

runs started: 1189; runs in flight at t=300 s with a refund still to come: 44
refunds executed after the flip, flag checked at admission: 50 (last at t=339 s)
refunds executed after the flip, flag checked at execution: 7 (last at t=307 s)

flag store unreachable from t=200 s; bad release from t=240 s makes half of later refunds wrong
refund calls after t=200 s: 812 (355 wrong, 457 correct)
on read error                       wrong executed  correct blocked
fail open (treat as enabled)                   355                0
fail closed (treat as disabled)                  0              457

In the first block of the output, 44 runs were in flight at the flip with a refund still to come. Checked at admission, 50 refunds ran after the flip, the last 39 seconds later. That count includes some runs admitted in the polling gap before a replica saw the flip. Checked at execution, 7 ran, all within 7 seconds, which is the remaining polling delay. The difference is the in-flight work.

Conditions that make a kill switch work

• A known propagation delay. The switch is only as fast as the slowest reader. A 10-second poll makes the switch effective within about 10 seconds. A configuration cached for five minutes does not. Measure the delay from the flip to the last replica that obeys it.
• A chosen default when the switch cannot be read. If the store holding the switch is unreachable, the executor must either proceed (fail open) or refuse (fail closed). The second block of the output counts the consequences when the store is down while the bad behavior is active: failing open lets all 355 wrong refunds through, and failing closed blocks 457 correct ones to prevent them. Choose per function. A tool whose effects cannot be undone usually fails closed, and a read-only search usually fails open.
• An independent, one-step control path. Flipping it must not need a deployment, must not run through the model it disables, and should not share a dependency, such as the same credentials or datastore, with what is failing. Whoever is on call has the permission in advance, and each change is logged.
• Narrow scope. Per tool, per tenant, or per feature, so that the least disruptive switch is available. A single global switch is available only as a blunt action.
• Visible state and a way back. During an incident, someone must be able to read which switches are off, and the plan names who turns a switch on again and under what evidence. A switch left off continues to block legitimate work.

A kill switch stops future calls. It does not reverse emails already sent or refunds already issued. It also needs to be exercised before an incident, by flipping it in a controlled setting and measuring the delay and the failure default, because an untested switch is an assumption. In an incident it is one of the actions of containment.

References

• Pete Hodgson, Feature Toggles (aka Feature Flags)