Graceful Shutdown

Graceful shutdown gives a process time to finish accepted work before exit. In Kubernetes, hooks and process cleanup share a termination budget, while routing withdrawal proceeds alongside them.

The platform aims to allow orderly exit while retaining a way to force termination. The sequence below describes normal Kubernetes graceful deletion; failures and forced termination can interrupt it.

The sequence

1. Deletion starts termination. The kubelet begins the local shutdown process with the configured grace period.
2. Endpoint conditions change concurrently. Terminating endpoints normally leave regular Service routing, but propagation and existing connections can outlast the update.
3. The hook precedes the signal. With a nonzero grace period, a configured preStop hook runs before the container’s stop signal. It consumes the same budget; completing the hook does not start a fresh full grace period.
4. Remaining processes face forced termination. When the grace budget expires, the runtime can kill them. Kubernetes documents a small one-off extension if a preStop hook is still running; do not plan cleanup around that exception.

Deletion-driven endpoint updates do not require a dedicated readiness probe. They also do not guarantee all traffic has stopped: publishNotReadyAddresses changes ready publication, and some proxies can use serving terminating endpoints when all available endpoints are terminating. Direct Pod access and existing connections need separate consideration.

Check which stop signal the runtime sends. Many runtimes honor the image’s STOPSIGNAL instead of SIGTERM. Verify that the application’s main process receives and handles the configured signal.

What your application has to do

Application code must coordinate its routing, acceptance of new work and cleanup within the remaining budget. Four responsibilities follow.

1. Handle the configured signal. An ignored signal can leave the process doing work until forced termination, without entering its cleanup path.
2. Stage withdrawal and acceptance. Coordinate readiness or discovery withdrawal with a bounded serving interval where needed. Then stop accepting new requests or pulling queue work.
3. Finish what is in flight. This is a separate action from the previous one, and conflating them is the most common bug in shutdown code: a process that closes its listener and exits immediately drops the requests it had already accepted.
4. Be safe to interrupt anyway. KILL always remains possible — a node can fail outright — so anything that must not happen twice needs idempotency rather than a clean exit.

Point 4 is the one to internalize, because it reframes the whole exercise. Graceful shutdown reduces how often you interrupt work; it never removes the possibility. Correctness under abrupt termination is a separate property that a grace period does not provide, and a system that depends on clean exits will eventually meet one that is not.

The grace period is a number you choose

Budget for preStop, routing propagation, in-flight work and cleanup with margin. For an illustrative sequential path, 5 seconds of withdrawal, 20 seconds to finish work and 10 seconds to flush already total 35 seconds. A normal 30-second budget does not cover that path. Validate overlap and actual durations for the implementation; abrupt interruption remains possible.

A kubelet or runtime-management restart can cause termination to retry with the original grace period. A different, feature-specific exception is the RestartAllContainers action under RestartAllContainersOnContainerExits: where supported and enabled, its fast in-place Pod restart skips preStop and the configured grace period. Do not generalize that exception to ordinary container restarts; check the cluster’s feature and version support.

Two assumptions that do not hold

Regular containers have no guaranteed stop order. Native sidecars—init containers with restartPolicy: Always—have a different contract: kubelet waits for main containers to finish and then stops the sidecars in reverse specification order. Grace expiry can still force the remaining containers to terminate. A proxy running as a regular container does not gain this ordering merely by being called a sidecar.

Traffic withdrawal need not finish before the signal arrives. Endpoint changes and local termination run concurrently. Balancer propagation, cached discovery and established connections can leave requests arriving after shutdown starts.

A readiness transition followed by a bounded serving interval can reduce this race; measure it within the total grace budget rather than treating a sleep as proof of convergence. Client retries with backoff need selected retryable failures, safe repetition and attempt/time limits. A write may have committed before its response was lost, so retry it only with an idempotency contract or evidence that it was not applied. Retries can reduce residual failures but do not guarantee success.

Finally, the frequency is worth keeping in view. On a container platform this path runs constantly — every rollout, every scale-down, every node drain — so shutdown is not an edge case exercised at release time but ordinary behavior that happens many times a day. A service that loses a few requests each time it stops is losing them continuously. See Ephemeral by Default for the surrounding lifecycle and partial failure for why a local recovery step has to be designed against the failure it will actually meet.

References

• Kubernetes: Pod Lifecycle
• Liveness, Readiness, and Startup Probes
• Kubernetes: EndpointSlices
• RFC 9110: Idempotent Methods