Kubernetes Controllers and Workload Lifecycle

Running one container is a command. Keeping a service made of several containers running for months is a different problem, because hosts fail, processes crash, new versions arrive while the old version is serving traffic, and nobody is watching at the moment each of those things happens. Kubernetes addresses this problem with a specific design: you declare what you want, and a set of independent programs called controllers repeatedly compare what you declared with what exists and act on the difference. Understanding that design is what separates reading Kubernetes manifests from being able to predict what a cluster will do, explain what it did, and tell whether it is stuck.

What this post assumes. From Containers: Images, Isolation, and Resource Boundaries it assumes you know that a container is a process with a filesystem from an image, restricted views from namespaces, and limits from control groups, and that a container exit code or a memory-limit kill is evidence you can read. This post does not repeat those mechanisms; it uses them. It also assumes general computer science knowledge: processes, client-server APIs, eventual consistency at the level of “two copies of state can disagree for a while”, and basic YAML. Everything specific to Kubernetes (objects, controllers, Pods, ReplicaSets, Deployments, probes, scheduling, requests and limits) is explained here.

What you should be able to do afterward. Explain desired-state reconciliation and why it tolerates missed events; describe what a Pod, ReplicaSet, and Deployment each own and how a rolling update proceeds; configure and reason about readiness, liveness, and startup probes; explain how the scheduler places a Pod and why a Pod stays Pending; predict what happens for a Pod crash, a node loss, a bad rollout, and a controller outage; and diagnose each from the objects’ status and events.

How the examples were produced. The traces below come from a small scratch model of reconciliation that is a teaching aid, not Kubernetes, and simplifies timing, networking, and many real rules. No Kubernetes cluster and no kubectl were available for this post, so the manifest and the kubectl commands are labeled as not executed. The behavior they describe is taken from the Kubernetes documentation cited in the references, read at the time of writing. Kubernetes behavior and defaults change between versions, so check the documentation for the version you operate.

Desired-state reconciliation

Intent and observation are stored separately

Kubernetes exposes its state through an API server. Everything you create, such as a Deployment, a Pod, or a Service, is an object stored through that API. Most objects have a spec that records what is wanted, and a status that records what components have observed. A Deployment’s spec might say three replicas of a given Pod template, and its status reports how many replicas exist, how many are available, and which generation of the spec the controller has processed. The two are written by different parties: users and automation write the spec, and controllers and kubelets write the status. That separation is what lets the system notice a gap between them. The desired state entry in this series’ glossary describes the same split and its caveats, including that the status can lag the latest request and that an accepted request is not proof that execution is feasible.

A controller is a loop, not an event handler

The Kubernetes documentation describes a controller as a non-terminating loop that regulates the state of a system, comparable to a thermostat that measures temperature and switches equipment toward a set point. The design has several consequences. A controller watches the objects relevant to its responsibility, compares desired and observed state, and takes action to reduce the difference. Many built-in controllers do not do the work themselves. The Job controller, for example, asks the API server to create Pods, and other components (the scheduler and the kubelets) act on those objects. Some controllers act on external systems, such as an autoscaler that provisions nodes. Kubernetes uses many small controllers rather than one monolithic loop; the documentation states that simple controllers are preferable to a single set of interlinked loops, and that several controllers can manage the same kind of object (Deployments and Jobs both create Pods) by using labels to identify what each owns. Built-in controllers run together in the kube-controller-manager, and the control plane can be run resiliently so that another instance takes over if one fails. See the control loop entry for how loops avoid fighting over objects and why loop activity alone is not evidence of health.

The word “loop” matters. A handler that reacts to an event (“a Pod was deleted, create one”) would need to receive every event exactly once to be correct. A loop that compares the full desired and observed states needs no such guarantee. Kubernetes’ list-and-watch API supports this: a client lists objects to get a consistent starting point with a resourceVersion, then watches for changes from that version. The resourceVersion is an opaque string that clients must not interpret. The watch history is finite, and if a client asks to resume from a version that is too old, the API server returns 410 Gone, and the client must list again and resume from a fresh version. In other words, the protocol is designed so that a controller that missed events recovers by re-reading state, not by replaying a lost event stream. This property is called level-triggered operation, as opposed to edge-triggered: the controller reacts to the level of the difference, not to the edge of an event.

Ownership ties the loops together

Controllers form chains through ownership. A Deployment owns ReplicaSets, and a ReplicaSet owns Pods. Each owned object records its owner in metadata.ownerReferences. Ownership serves two purposes. The ReplicaSet controller uses it to know which Pods it manages. And the garbage collector uses it to delete dependents when an owner is deleted: by default, deleting a Deployment deletes the ReplicaSets and Pods it owns in the background, and --cascade=orphan leaves them running without an owner. A label selector identifies candidate objects, but the ReplicaSet documentation adds a subtle rule: a ReplicaSet acquires any matching Pod that has no controller owner, even one it did not create. Overlapping selectors on workload controllers therefore produce ambiguous management, and label changes on live Pods change what a controller sees.

A traced model

To make the loops concrete, the traces below come from a small scratch model written for this post: a store standing in for the API server, a Deployment controller, a ReplicaSet controller, a scheduler, and a kubelet, each re-reading the whole store on every tick. The model gives each Pod a 400 millicore request on 1000 millicore nodes, so a node holds two Pods. The model’s full source follows, and each trace can also be followed by hand from the rules in the text. Treat the traces as illustrations of the stated rules, not as measurements of Kubernetes: real controllers use caches and work queues, real Pods have many more states, and real timing is asynchronous and variable. In each line, spec is a ReplicaSet’s desired count, pods is how many Pod objects exist, and ready is how many are Ready.

"""A toy model of Kubernetes-style reconciliation. Written for this post; it is NOT Kubernetes.
It models: an API store, a Deployment controller, a ReplicaSet controller, a scheduler, and a kubelet.
Every controller re-reads the whole store each tick (level-triggered): it compares desired and observed
state and fixes the difference, regardless of which events it might have missed."""
import itertools

class Store:
    def __init__(self):
        self.rv = 0; self.pods = {}; self.rsets = {}; self.dep = None; self.nodes = {}; self.events = []
        self.ids = itertools.count(1)
    def touch(self): self.rv += 1

class Pod:
    def __init__(self, store, rs):
        self.name = f"{rs.name}-{next(store.ids)}"; self.rs = rs.name; self.version = rs.version
        self.node = None; self.phase = "Pending"; self.age = 0; self.ready = False; self.restarts = 0; self.deleting = False

class RS:
    def __init__(self, version): self.name = f"rs-{version}"; self.version = version; self.replicas = 0

def dep_controller(s, healthy_versions):
    d = s.dep
    new = s.rsets.setdefault(d["version"], RS(d["version"]))
    olds = [r for v, r in s.rsets.items() if v != d["version"]]
    live = lambda r: [p for p in s.pods.values() if p.rs == r.name and not p.deleting]
    available = sum(1 for r in s.rsets.values() for p in live(r) if p.ready)
    total = sum(r.replicas for r in s.rsets.values())
    max_total = d["replicas"] + d["max_surge"]; min_avail = d["replicas"] - d["max_unavailable"]
    # scale the new ReplicaSet up within the surge budget
    room = max_total - total
    if new.replicas < d["replicas"] and room > 0:
        new.replicas = min(d["replicas"], new.replicas + room)
    # old ReplicaSets: pods that are not ready are removed first (removing them costs no availability)
    for r in olds:
        unready = sum(1 for p in live(r) if not p.ready)
        r.replicas -= min(r.replicas, unready)
    total = sum(r.replicas for r in s.rsets.values())
    room = max_total - total
    if new.replicas < d["replicas"] and room > 0:
        new.replicas = min(d["replicas"], new.replicas + room)
    # then scale old ReplicaSets down within the availability budget
    spare = available - min_avail
    for r in olds:
        if spare > 0 and r.replicas > 0:
            cut = min(spare, r.replicas); r.replicas -= cut; spare -= cut
    if new.replicas > d["replicas"]: new.replicas = d["replicas"]

def rs_controller(s):
    for r in s.rsets.values():
        mine = [p for p in s.pods.values() if p.rs == r.name and not p.deleting]
        for _ in range(r.replicas - len(mine)):
            p = Pod(s, r); s.pods[p.name] = p; s.touch()
        extra = len(mine) - r.replicas
        # prefer removing pods that are not ready, then newest
        for p in sorted(mine, key=lambda p: (p.ready, -p.age))[:max(extra, 0)]:
            p.deleting = True

def scheduler(s, cpu_request=400):
    for p in sorted(s.pods.values(), key=lambda p: p.name):
        if p.node or p.deleting: continue
        used = {n: sum(cpu_request for q in s.pods.values() if q.node == n and not q.deleting) for n in s.nodes}
        feasible = [n for n in s.nodes if s.nodes[n]["up"] and used[n] + cpu_request <= s.nodes[n]["cpu"]]
        if feasible: p.node = min(feasible, key=lambda n: used[n]); p.phase = "ContainerCreating"
        elif "FailedScheduling" not in [e[1] for e in s.events if e[2] == p.name]:
            s.events.append((s.t, "FailedScheduling", p.name))

def kubelet(s, healthy_versions, ready_after=2):
    for name, p in list(s.pods.items()):
        if p.deleting: del s.pods[name]; s.touch(); continue
        if p.node and not s.nodes[p.node]["up"]: del s.pods[name]; s.touch(); continue   # node lost
        if not p.node: continue
        p.age += 1
        if p.age >= 1: p.phase = "Running"
        if p.version in healthy_versions: p.ready = p.age >= 1 + ready_after
        else: p.ready = False; p.restarts += (1 if p.age % 2 == 0 else 0)             # crash, back off, crash...

def show(s):
    parts = []
    for v, r in sorted(s.rsets.items()):
        ps = [p for p in s.pods.values() if p.rs == r.name and not p.deleting]
        parts.append(f"{v}: spec={r.replicas} pods={len(ps)} ready={sum(p.ready for p in ps)}")
    pend = sum(1 for p in s.pods.values() if not p.node)
    return " | ".join(parts) + (f" | unscheduled={pend}" if pend else "")

def run(s, ticks, healthy, label, hook=None):
    print(f"--- {label}")
    for _ in range(ticks):
        s.t += 1
        if hook: hook(s)
        dep_controller(s, healthy); rs_controller(s); scheduler(s); kubelet(s, healthy)
        print(f" t={s.t:02d} {show(s)}")

s = Store(); s.t = 0
s.nodes = {f"node-{i}": {"cpu": 1000, "up": True} for i in "abc"}      # each node fits two 400m pods
s.dep = {"replicas": 3, "version": "v1", "max_surge": 1, "max_unavailable": 1}
run(s, 5, {"v1", "v2"}, "1. create a Deployment with 3 replicas")

def kill_one(s):
    if s.t == s.kill_at: p = next(iter(s.pods.values())); p.deleting = True
s.kill_at = s.t + 1
run(s, 4, {"v1", "v2"}, "2. a pod is deleted; the ReplicaSet notices a difference and replaces it", kill_one)

s.dep["version"] = "v2"
run(s, 7, {"v1", "v2"}, "3. template changes to v2 (healthy); rolling update with maxSurge=1, maxUnavailable=1")

s.dep["version"] = "v3"
run(s, 6, {"v1", "v2"}, "4. template changes to v3, which never becomes ready (v3 not in the healthy set)")
print(" -> Deployment condition would be reported after progressDeadlineSeconds; nothing rolls back by itself.")
s.dep["version"] = "v2"
run(s, 5, {"v1", "v2"}, "5. an operator sets the template back to v2 (a rollback is just another template change)")

def lose_node(s):
    if s.t == s.lose_at: s.nodes["node-a"]["up"] = False
s.lose_at = s.t + 1
run(s, 5, {"v1", "v2"}, "6. node-a fails; its pods vanish and are recreated elsewhere", lose_node)

def outage(s, ticks):
    print("--- 7. the controllers stop for 3 ticks while two pods are deleted; then they come back")
    for p in list(s.pods.values())[:2]: p.deleting = True
    for _ in range(ticks):
        s.t += 1; kubelet(s, {"v1", "v2"})                  # only the node agent runs
        print(f" t={s.t:02d} (controllers down) {show(s)}")
outage(s, 3)
run(s, 4, {"v1", "v2"}, "7b. controllers restart: they re-read the store and repair the difference")
s.dep["replicas"] = 6
run(s, 4, {"v1", "v2"}, "8. scale to 6 replicas with only two nodes left (4 pods of capacity)")
print(" events:", [(t, r, n) for t, r, n in s.events])

The first two scenarios show the basic behavior. The trace for creating a Deployment with three replicas, then deleting one Pod:

--- 1. create a Deployment with 3 replicas
 t=01 v1: spec=3 pods=3 ready=0
 t=02 v1: spec=3 pods=3 ready=0
 t=03 v1: spec=3 pods=3 ready=3
 t=04 v1: spec=3 pods=3 ready=3
 t=05 v1: spec=3 pods=3 ready=3
--- 2. a pod is deleted; the ReplicaSet notices a difference and replaces it
 t=06 v1: spec=3 pods=3 ready=2
 t=07 v1: spec=3 pods=3 ready=2
 t=08 v1: spec=3 pods=3 ready=3
 t=09 v1: spec=3 pods=3 ready=3

Notice what the replacement does and does not do. The ReplicaSet does not restart the deleted Pod; it creates a new one, which has a new name and goes through scheduling and readiness again. In the model, ready falls to 2 for two ticks while the replacement starts. The same is true in Kubernetes: the Pod documentation says a replaced Pod has a new identity and the original is never rescheduled to a different node, which is why Pods are described as ephemeral and why durable behavior comes from the controller above them.

The seventh scenario tests the level-triggered claim. The controllers stop for three ticks while two Pods are deleted, then restart. Nothing in the model records which events were missed:

--- 7. the controllers stop for 3 ticks while two pods are deleted; then they come back
 t=33 (controllers down) v1: spec=0 pods=0 ready=0 | v2: spec=3 pods=1 ready=1 | v3: spec=0 pods=0 ready=0
 t=34 (controllers down) v1: spec=0 pods=0 ready=0 | v2: spec=3 pods=1 ready=1 | v3: spec=0 pods=0 ready=0
 t=35 (controllers down) v1: spec=0 pods=0 ready=0 | v2: spec=3 pods=1 ready=1 | v3: spec=0 pods=0 ready=0
--- 7b. controllers restart: they re-read the store and repair the difference
 t=36 v1: spec=0 pods=0 ready=0 | v2: spec=3 pods=3 ready=1 | v3: spec=0 pods=0 ready=0
 t=37 v1: spec=0 pods=0 ready=0 | v2: spec=3 pods=3 ready=1 | v3: spec=0 pods=0 ready=0
 t=38 v1: spec=0 pods=0 ready=0 | v2: spec=3 pods=3 ready=3 | v3: spec=0 pods=0 ready=0
 t=39 v1: spec=0 pods=0 ready=0 | v2: spec=3 pods=3 ready=3 | v3: spec=0 pods=0 ready=0

During the outage the workload runs below its desired count and nothing repairs it, which is the cost of a controller being unavailable. After the controllers return, they compute the gap from the current store and repair it in one step. They did not need the missed deletion events. The model also shows why controller health is a thing to monitor. A Deployment whose controller is down still has a spec that says three replicas and a status that stops changing, so stale status is itself evidence.

Two limits of this guarantee should be stated. Convergence is not instantaneous or guaranteed: the loop can only act within what the rest of the system allows, so a Pod that cannot be scheduled or whose image cannot be pulled stays that way. And reconciliation repairs differences from the spec, so if someone edits the spec, for example by changing the replica count by hand, the controller treats the new value as the target. A repository file is a separate source of intent that needs its own process to keep the API objects in sync.

Pods and deployment controllers

The Pod as the unit of execution

A Pod groups one or more tightly coupled containers that are scheduled together onto one node. Containers in a Pod share a network namespace, so they have one IP address and port space and can talk over localhost, and they can share declared volumes. The previous post explained the namespaces behind that. A Pod is an execution attempt, not a durable record. If a node is lost or the Pod is deleted, the object is gone; a controller above it creates a new one.

A Pod’s phase summarizes where it is in its lifecycle: Pending (accepted but not all containers running, which includes the time to be scheduled and to pull images), Running (bound to a node with at least one container running or restarting), Succeeded or Failed (all containers terminated, with all exit codes zero for Succeeded), and Unknown. The phase is coarse, so use finer signals. Each container has a state of Waiting (with a reason such as ContainerCreating or ImagePullBackOff), Running, or Terminated (with an exit code, reason, and times). The Pod’s conditions record PodScheduled, Initialized, ContainersReady, and Ready. Ready is the condition that service routing uses. A Pod can be Running and not Ready, which is the normal state while it initializes.

Within a Pod, the restartPolicy (default Always) tells the kubelet whether to restart a container that exits. Repeated failures cause the kubelet to wait longer between restarts, with a delay that starts small, doubles on each failure, and is capped at five minutes, and that resets after the container has run successfully for ten minutes according to the documentation. The status CrashLoopBackOff is how you see that: it names the waiting interval between restarts of a container that keeps failing, not a separate failure. Two consequences follow. The restart count is evidence that the container, not the Pod object, is failing, and kubectl logs --previous shows the previous container instance’s output. And a restart is local: it happens in the same Pod on the same node, with the same Pod identity, so a problem caused by the node or the Pod’s configuration persists across restarts.

ReplicaSets and Deployments

A ReplicaSet maintains a set of Pods. It has a Pod template, a label selector, and a replica count, and it creates or deletes Pods to match the count. The documentation recommends using Deployments instead of ReplicaSets directly unless you need custom update orchestration.

A Deployment manages ReplicaSets declaratively. When the Pod template changes, the Deployment creates a new ReplicaSet for the new template, identified by a pod-template-hash label, and gradually scales the new ReplicaSet up while scaling the old one down. The rollout is triggered only by changes to .spec.template. Changing the replica count scales the existing ReplicaSet and does not create a new revision. The Deployment keeps old ReplicaSets, up to revisionHistoryLimit (default 10), so that earlier revisions can be restored.

The default RollingUpdate strategy is governed by two parameters, each of which can be a number or a percentage. maxUnavailable (default 25%) bounds how many Pods may be unavailable during the update, and maxSurge (default 25%) bounds how many Pods above the desired count may exist. For a Deployment of three replicas with maxUnavailable: 1 and maxSurge: 1, as in the documentation’s example, at least two Pods stay available and up to four exist at once. These two numbers define a corridor: the Deployment controller can scale the new ReplicaSet up as long as the total stays within the surge limit, and scale the old ReplicaSet down only as long as available Pods stay above the unavailability limit. The Recreate strategy instead terminates all old Pods before creating new ones, which causes downtime and fits applications that cannot run two versions at once. minReadySeconds (default 0) requires a new Pod to be ready for a number of seconds before counting as available.

Availability is defined by readiness. A new Pod counts toward availability only once it is Ready. This ties the rollout to the readiness probe, and it is why a Deployment without a meaningful readiness probe cannot protect you: if Pods become Ready the moment their process starts, the rollout proceeds as fast as containers start, whatever state the application is in.

Reading a rollout in the model

The model implements the corridor with max_surge=1 and max_unavailable=1. Scenario 3 changes the template from v1 to a healthy v2:

--- 3. template changes to v2 (healthy); rolling update with maxSurge=1, maxUnavailable=1
 t=10 v1: spec=2 pods=2 ready=2 | v2: spec=1 pods=1 ready=0
 t=11 v1: spec=2 pods=2 ready=2 | v2: spec=2 pods=2 ready=0
 t=12 v1: spec=2 pods=2 ready=2 | v2: spec=2 pods=2 ready=1
 t=13 v1: spec=1 pods=1 ready=1 | v2: spec=2 pods=2 ready=2
 t=14 v1: spec=0 pods=0 ready=0 | v2: spec=3 pods=3 ready=2
 t=15 v1: spec=0 pods=0 ready=0 | v2: spec=3 pods=3 ready=2
 t=16 v1: spec=0 pods=0 ready=0 | v2: spec=3 pods=3 ready=3

Follow the arithmetic at the start of the rollout. Three Pods are ready, and the corridor allows a total of four and a minimum of two ready. The controller first adds one Pod to the new ReplicaSet (total four) and removes one from the old ReplicaSet because three ready Pods are one above the floor of two. The new Pod is not ready yet, so the floor holds the old ReplicaSet at two until new Pods become ready. Total ready never drops below two. The rollout is paced by how fast new Pods become ready, not by a timer, which is exactly why the quality of the readiness signal decides the safety of the rollout. This is the mechanism that makes the canary and progressive approaches from earlier in this series possible at the level of Pods, though a Deployment alone does not shift traffic by percentage or compare metrics. It only controls how many Pods of each version exist and are ready.

Scenario 4 changes the template to a version whose Pods never become ready:

--- 4. template changes to v3, which never becomes ready (v3 not in the healthy set)
 t=17 v1: spec=0 pods=0 ready=0 | v2: spec=2 pods=2 ready=2 | v3: spec=1 pods=1 ready=0
 t=18 v1: spec=0 pods=0 ready=0 | v2: spec=2 pods=2 ready=2 | v3: spec=2 pods=2 ready=0
 t=19 v1: spec=0 pods=0 ready=0 | v2: spec=2 pods=2 ready=2 | v3: spec=2 pods=2 ready=0
 t=20 v1: spec=0 pods=0 ready=0 | v2: spec=2 pods=2 ready=2 | v3: spec=2 pods=2 ready=0
 t=21 v1: spec=0 pods=0 ready=0 | v2: spec=2 pods=2 ready=2 | v3: spec=2 pods=2 ready=0
 t=22 v1: spec=0 pods=0 ready=0 | v2: spec=2 pods=2 ready=2 | v3: spec=2 pods=2 ready=0

The rollout stops with two ready old Pods serving and two not-ready new Pods that will not become ready. The corridor did its job: it prevented the broken version from replacing the working one, and the service kept two of three replicas, so capacity is reduced but not gone. What the model does not do is repair itself, and neither does Kubernetes. The documentation says that when a Deployment stalls, Kubernetes reports it and takes no other action: after progressDeadlineSeconds (default 600) without progress, the Progressing condition becomes false with reason ProgressDeadlineExceeded, and nothing rolls back automatically. The documented causes of a stalled Deployment include insufficient quota, readiness probe failures, image pull errors, limit ranges, and application misconfiguration. The deadline measures stalled progress, not total rollout time. A policy for what happens next (alert, pause, roll back, or fix forward) is a decision the platform must make, and that is usually implemented by the delivery tooling that watches these conditions rather than by the Deployment itself.

Scenario 5 shows that a rollback is another reconciliation. The operator sets the template back to the previous version, which in Kubernetes can be done with kubectl rollout undo against a retained revision:

--- 5. an operator sets the template back to v2 (a rollback is just another template change)
 t=23 v1: spec=0 pods=0 ready=0 | v2: spec=3 pods=3 ready=2 | v3: spec=0 pods=0 ready=0
 t=24 v1: spec=0 pods=0 ready=0 | v2: spec=3 pods=3 ready=2 | v3: spec=0 pods=0 ready=0
 t=25 v1: spec=0 pods=0 ready=0 | v2: spec=3 pods=3 ready=3 | v3: spec=0 pods=0 ready=0

In the model, the controller removes the not-ready v3 Pods first because removing them costs no availability, and it scales v2 back up. This particular rule, removing unready Pods of an old ReplicaSet before spending availability budget, is a modeling choice made so that the example would recover; how the real Deployment controller orders these steps in every case was not verified here, so do not rely on the model for that detail. The general point stands: the cluster does not remember the failed attempt as a state to undo. It sees a new desired template and reconciles toward it.

A rollback is also only as good as its compatibility with the data and interfaces that other components now use. Returning a Deployment to an earlier template restores earlier code. It does not reverse a schema migration or unsend messages the new version produced, which is why the compatibility discipline from the progressive delivery post is a precondition for a safe rollback.

A manifest, with the declarations in one place

The manifest below collects the settings discussed so far. It was written for this post and was not applied to a cluster, since none was available. The image reference uses a placeholder digest, and the structure follows the Deployment and probe documentation.

# Illustrative manifest. Not executed. Names and the digest are placeholders.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
  minReadySeconds: 5
  selector:
    matchLabels:
      app: orders-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: app
          image: registry.example.com/team/orders-api@sha256:<digest>
          ports:
            - containerPort: 8080
          resources:
            requests: { cpu: 250m, memory: 256Mi }
            limits:   { memory: 256Mi }
          startupProbe:
            httpGet: { path: /started, port: 8080 }
            periodSeconds: 5
            failureThreshold: 24
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet: { path: /alive, port: 8080 }
            periodSeconds: 10
            failureThreshold: 3

Read the manifest as a set of statements, each answered by a different component. The selector and template labels tell the controllers which Pods belong together. The strategy defines the rollout corridor. The requests tell the scheduler how much to reserve and the limits tell the kubelet and runtime what to enforce through cgroups. The probes tell the kubelet how to judge the container. The image digest, as the earlier posts explained, identifies the exact bytes.

Probes

Three questions, three probes

The kubelet runs probes against containers. Each probe answers a different question, and confusing them is a frequent cause of outages.

Probe	Question	On failure	Typical use
Readiness	Can this instance serve requests now?	The Pod is marked not Ready and removed from Service endpoints; the container is not restarted	Warm-up, temporary overload, waiting for a dependency the instance cannot work without
Liveness	Is the container stuck in a state a restart would fix?	The container is killed and restarted according to restartPolicy	Detecting deadlocks
Startup	Has the application finished starting?	Treated like a liveness failure once the threshold is reached	Slow-starting applications; holds back liveness and readiness until it succeeds

A probe can use an HTTP GET (success for status codes from 200 to 399), a TCP connection, a command whose exit code zero means success, or a gRPC health check. The default timing parameters per the documentation are initialDelaySeconds 0, periodSeconds 10, timeoutSeconds 1, failureThreshold 3, and successThreshold 1. Readiness probes have a dedicated glossary entry that goes through endpoint publication details and the reasoning about dependencies.

Timing is arithmetic you can do in advance

With defaults, a failing check is noticed after about failureThreshold consecutive failures spaced periodSeconds apart, so roughly 30 seconds, plus the time to run the probe. This is a derived estimate from the parameters, not a measurement, and the exact timing depends on when within a period the failure begins. The same arithmetic sizes a startup probe: in the manifest above, failureThreshold: 24 with periodSeconds: 5 allows about 120 seconds for the application to start before the container is killed. Choose that budget from measured startup times under realistic conditions, with margin, instead of from a guess, and recheck it when the application changes. If startup occasionally takes much longer under load, a budget that is too small turns a slow start into a restart loop with increasing delays.

What each probe should check

Readiness should reflect whether this instance can do useful work now. That includes initialization and local resources. Whether to include a dependency is a judgment: if the instance cannot serve anything useful without the dependency, include it, but remember that a shared dependency failing makes every replica go not-ready at once, and the service becomes unavailable instead of degraded. If the instance can serve a useful subset without it, excluding the dependency from readiness lets it keep serving that subset. The readiness probe entry works through this choice.

Liveness should answer only whether a restart could help. The Kubernetes documentation warns that overly aggressive liveness probes can cause unnecessary restarts and cascading failures. The mechanism is easy to follow. Under heavy load, responses slow, a probe times out, the container is restarted and loses its warm state and in-flight work, its traffic moves to the remaining replicas, which slow down in turn, and their probes begin failing. A liveness check that depends on load or on external dependencies converts slowness into lost capacity. Prefer a liveness check that tests only the process’s own ability to make progress, with thresholds that tolerate transient slowness, and use readiness for load-related withdrawal. Some services need no liveness probe at all, because the process exits when it is broken.

Probes and rollout interact. The Deployment’s notion of an available Pod depends on readiness, so a readiness endpoint that always returns success removes the rollout’s safety property, and one that fails for reasons outside the instance can stall a rollout without any problem in the new version. Test the endpoint’s behavior during startup, under load, and during shutdown.

Shutdown is part of the same design

When a Pod is deleted, whether by a rollout, a scale-down, or a node drain, termination and routing withdrawal proceed together. The kubelet runs any preStop hook, then signals the containers to stop, and force-kills them if they are still running when the grace period (terminationGracePeriodSeconds, default 30 seconds) expires. The hook and the cleanup share that same budget. The Pod is removed from regular Service endpoints, but propagation of that change takes time, so requests can still arrive while the process is stopping. The graceful shutdown entry gives the details and the sizing arithmetic. The consequence for this post is that every rolling update is also a sequence of shutdowns, so a service that drops in-flight requests on SIGTERM is losing requests on every deployment. Because a node can fail outright, anything that must not happen twice needs idempotency rather than a clean exit.

Scheduling and resource declarations

How the scheduler places a Pod

The scheduler assigns Pods that have no node. Its default behavior has two steps followed by a binding. In filtering, it removes nodes that cannot run the Pod, for example because they lack enough unreserved resources for its requests. In scoring, it ranks the remaining feasible nodes and picks the highest. It then records the choice by binding the Pod to the node through the API server. Ties are broken randomly. If no node passes filtering, the Pod stays Pending and a FailedScheduling event is recorded, and it remains unscheduled until conditions change, for example when capacity is added or other Pods leave. The scheduler is another controller in the sense used above: it watches for Pods without nodes and acts on the difference.

One detail catches many newcomers. The scheduler compares requests, not actual usage. A node with idle CPUs can be “full” if the sum of the requests of its Pods already equals its allocatable capacity, and a node that looks busy can accept more Pods if its Pods requested little. Requests are what you ask the platform to reserve for placement. Whether a Pod actually gets that much depends on enforcement and contention.

Requests, limits, and the cgroups beneath them

Requests and limits are declared per container. The documentation describes requests as used by the scheduler and as the amount the container is guaranteed, and limits as enforced by the kubelet and the container runtime, which implement them with the cgroup controllers described in the previous post. Exceeding a CPU limit leads to throttling; exceeding a memory limit may lead to the container being killed by the kernel for running out of memory, shown as OOMKilled. If you specify a limit and no request, Kubernetes sets the request equal to the limit. CPU is measured in cores or millicores (250m is a quarter core), and memory in bytes with suffixes such as Mi.

The declarations determine a Pod’s quality-of-service class. A Pod is Guaranteed if every container has CPU and memory requests equal to limits, BestEffort if no container has any CPU or memory request or limit, and Burstable otherwise. The class matters when a node runs short of resources. The kubelet’s node-pressure eviction watches signals such as available memory and filesystem space, and evicts Pods when thresholds are crossed. It ranks candidates mainly by how far their usage exceeds their requests, then by Pod priority, and the QoS class correlates with that order, with BestEffort first and Guaranteed last. An evicted Pod ends in the Failed phase with reason Evicted, which is different from an OOM kill of a container inside a Pod that remains. The documentation also states that node-pressure eviction does not respect PodDisruptionBudgets, and that hard thresholds terminate Pods without honoring the grace period.

These facts support concrete advice. A memory request below real steady-state usage lets the scheduler pack a node beyond what it can hold, which shows up later as evictions. A memory limit below the real peak shows up as OOMKilled restarts under load. A CPU limit can add tail latency through throttling even when the node is idle; the previous post’s quota model explains why. Setting a memory request equal to the memory limit, as in the manifest, makes memory behavior predictable at the cost of unused headroom, while leaving the CPU limit unset avoids throttling and relies on requests and scheduling for fairness. Whether to cap CPU is a trade-off between neighbor protection and latency, and it depends on how much contention your nodes see; measure before deciding. The noisy neighbor entry describes what limits protect and what they do not.

The scheduler in the model

Scenario 8 scales the Deployment to six replicas on a cluster whose remaining two nodes each fit two Pods:

--- 8. scale to 6 replicas with only two nodes left (4 pods of capacity)
 t=40 v1: spec=0 pods=0 ready=0 | v2: spec=6 pods=6 ready=3 | v3: spec=0 pods=0 ready=0 | unscheduled=2
 t=41 v1: spec=0 pods=0 ready=0 | v2: spec=6 pods=6 ready=3 | v3: spec=0 pods=0 ready=0 | unscheduled=2
 t=42 v1: spec=0 pods=0 ready=0 | v2: spec=6 pods=6 ready=4 | v3: spec=0 pods=0 ready=0 | unscheduled=2
 t=43 v1: spec=0 pods=0 ready=0 | v2: spec=6 pods=6 ready=4 | v3: spec=0 pods=0 ready=0 | unscheduled=2
 events: [(40, 'FailedScheduling', 'rs-v2-15'), (40, 'FailedScheduling', 'rs-v2-16')]

The ReplicaSet did its job: it created six Pods, because the spec says six. Two Pods cannot be placed, so they stay unscheduled with FailedScheduling events. The Deployment’s spec is satisfied at the object level but not at the availability level, which is the same distinction Kubernetes status fields draw. This is the situation in which an operator reading kubectl get pods sees Pending Pods and kubectl describe shows the scheduler’s explanation. The Kubernetes debugging guide lists insufficient resources and hostPort conflicts among the common causes. The remedy belongs to a different layer: reduce requests if they are inflated, add capacity, or accept lower replica counts, and an autoscaler that adds nodes in response to unschedulable Pods is yet another controller acting on the same signal.

Failure and recovery

With the pieces in place, it is possible to walk through failures and say which loop responds, what it does, and what it cannot do. The table summarizes the cases. Behavior marked as model was seen in the traces above; the rest comes from the documentation.

Failure	Loop that responds	What happens	What it does not cover
Container process exits	Kubelet, per restartPolicy	Container restarted in the same Pod with exponentially increasing delay (CrashLoopBackOff)	A fault in the node, image, or configuration persists across restarts
Container exceeds memory limit	Kernel, then kubelet	Container killed (OOMKilled) and restarted	Does not tell you whether the limit or the application is wrong
Pod deleted	ReplicaSet controller	New Pod created and scheduled (model scenario 2)	Work in the old Pod is lost; external side effects may have happened
Node lost	Control plane and ReplicaSet controller	Pods on the node are eventually marked for deletion after a timeout and replaced elsewhere; the model removes them immediately	Capacity on other nodes must exist; time to detect is not zero
Node under resource pressure	Kubelet eviction	Pods evicted in ranked order, replaced by their controllers	Does not honor PodDisruptionBudgets; uses requests to rank
No node fits a Pod	Scheduler	Pod stays Pending with FailedScheduling (model scenario 8)	Capacity must come from elsewhere
Bad rollout	Deployment controller	Rollout stalls inside the surge and unavailability corridor (model scenario 4)	No automatic rollback; reported after progressDeadlineSeconds
Controller unavailable	Another control-plane instance, if present	Repair pauses until a controller runs; then re-reads state (model scenario 7)	Stale status is the symptom; monitor controller health
Voluntary disruption (drain, scale-down)	Eviction API with PodDisruptionBudget	Evictions limited so that a replicated application keeps a minimum available	Does not cover involuntary failures, direct deletion, or rolling updates

A few of these deserve extra comment. The PodDisruptionBudget limits how many Pods of an application can be down at once because of voluntary disruptions that go through the eviction mechanism, such as a node drain. The documentation lists what it does not protect against: involuntary disruptions such as hardware failure and out-of-resource evictions, direct deletion of Pods or Deployments, and rolling updates, which are governed by the Deployment’s own parameters. A budget of minAvailable: 2 for three replicas combined with a Deployment strategy that allows only two available Pods leaves no room to drain a node at the same moment as a rollout, so set budgets with the rollout corridor in mind. The documentation also recommends setting unhealthyPodEvictionPolicy: AlwaysAllow so that Pods that are not healthy cannot block a node drain forever.

Recovery by replacement has a limit that deserves to be explicit. Replacing a Pod restores the count of running instances. It does not restore state, and it does not undo effects. A job that wrote half its output before the node died may run again from the start, so processing that must not repeat needs identity and idempotency. State that must outlive a Pod needs storage with its own lifecycle, such as a PersistentVolumeClaim, whose reclaim policy decides whether the underlying data survives deletion. The next post, Kubernetes Networking, Storage, and Stateful Workloads, develops those topics. Finally, recovery loops can interact badly with overload: replacement Pods compete for the same scarce capacity, and restart or retry behavior can amplify a failure, which is a cascading failure pattern worth testing for.

Diagnosing a workload

The status and events that Kubernetes records make diagnosis a matter of reading evidence in order from the outermost controller inward. The sequence below follows the object chain. The commands were not executed for this post; they come from the Kubernetes documentation’s debugging and rollout guidance.

# Not executed in this post.

# 1. Did the Deployment controller see the latest spec, and is the rollout progressing?
kubectl get deployment orders-api -o wide
kubectl rollout status deployment/orders-api
kubectl describe deployment orders-api          # Conditions: Progressing, Available; events

# 2. Which ReplicaSets exist, and what are their desired and ready counts?
kubectl get replicaset -l app=orders-api

# 3. For Pods: phase, container state, restarts, node, and the reasons in Events.
kubectl get pods -l app=orders-api -o wide
kubectl describe pod <pod-name>                  # Conditions, Events (FailedScheduling, probe failures, pulls)

# 4. For a container that keeps failing: the previous instance's output.
kubectl logs <pod-name> --previous

# 5. If you need to undo a template change (history is kept per revisionHistoryLimit).
kubectl rollout history deployment/orders-api
kubectl rollout undo deployment/orders-api

Read the outputs with a model of who writes them. If a Deployment’s observedGeneration is behind its generation, the controller has not processed the latest spec, which points to the controller or API rather than the application. If the rollout has not progressed and the new ReplicaSet’s Pods are Pending, look at FailedScheduling events: the problem is capacity or constraints. If they are Waiting with an image-related reason, the problem is image access or the reference. If they are Running and not Ready, the container started but the readiness check fails, so look at the application and the probe configuration. If they restart repeatedly, check the previous logs and the exit code, then the memory limit evidence described in the previous post. Each branch corresponds to a different component, and the status tells you which component to look at first.

A diagnostic habit worth building is comparing the declared intent with the observed state before changing either. The Kubernetes debugging guide suggests, for a Pod that runs but misbehaves, fetching the object as the API server stores it and comparing it with the original manifest, because typos and incorrect nesting can cause fields to be ignored or defaulted. The object in the API is what controllers act on, so it is the authority on what the cluster believes it was asked to do.

Limits of the model and what to verify on a real cluster

The scratch model is a deliberately small version of a large system. It omits the API server’s consistency and admission machinery, caches and work queues, leader election, node heartbeats and the delay before a lost node’s Pods are removed, Pod priority and preemption, topology and affinity constraints, and the many ways real components retry. Its rollout algorithm resembles the documented corridor but differs in details, and the rollback ordering in scenario 5 is a modeling choice. Kubernetes documentation statements are as of the time of writing, and defaults and feature behavior vary by version and distribution; verify against your cluster’s version. No cluster was used, so no claim in this post comes from an observed Kubernetes run.

Several claims above are worth confirming on the cluster that will actually run a workload, because their numbers depend on the version, the nodes, and the application. The time from a Pod deletion to a Ready replacement, and how that changes when the scheduler cannot place it, is a property of the cluster. Whether a rollout to an image that fails its readiness probe stalls inside the corridor, and when ProgressDeadlineExceeded appears, can be read from the Deployment’s conditions. A memory limit below an application’s real peak leaves OOMKilled evidence in the container status. The number of failed requests when a running service under load receives a termination signal depends on the readiness transition and the grace period. A node drain behaves differently with and without a PodDisruptionBudget. Observations like these, not the documentation alone, should drive how workloads are sized and configured.

References

• Kubernetes documentation, Controllers
• Kubernetes documentation, Deployments
• Kubernetes documentation, ReplicaSet
• Kubernetes documentation, Pod Lifecycle
• Kubernetes documentation, Configure Liveness, Readiness and Startup Probes
• Kubernetes documentation, Kubernetes Scheduler
• Kubernetes documentation, Resource Management for Pods and Containers
• Kubernetes documentation, Pod Quality of Service Classes
• Kubernetes documentation, Node-pressure Eviction
• Kubernetes documentation, Disruptions
• Kubernetes documentation, Garbage Collection
• Kubernetes documentation, Kubernetes API Concepts
• Kubernetes documentation, Debug Pods