Nobody Is Running Your Manifest: How Kubernetes Reconciles Declared State

A team ships a change. The pipeline runs kubectl apply, the command prints deployment.apps/checkout configured, the exit code is zero, and the pipeline goes green. Forty minutes later a customer reports that the new pricing logic is not live. The old version is still serving every request, and no alert has fired anywhere.

Nothing failed, because nothing was ever running the manifest. (The scenario is illustrative, but the shape of it is the most common surprise for people arriving from imperative deployment tooling.) The exit code confirmed that the cluster accepted a description. What happens after that is a different system with different failure semantics, and this is what that system is.

Quotations below are from the Kubernetes documentation as read in September 2026. Field names and defaults are versioned and do change.

A manifest is a record of intent

The documentation’s own phrase is worth keeping: “a Kubernetes object is a ‘record of intent’ — once you create the object, the Kubernetes system will constantly work to ensure that the object exists.”

That intent lives in two fields that almost every object has. The spec is “a description of the characteristics you want the resource to have: its desired state.” The status is “the current state of the object, supplied and updated by the Kubernetes system and its components.” You write the first. The system writes the second.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 3          # what you want
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout  # what the created Pods will carry
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:1.9.2

The documentation walks through exactly this case. Set three replicas, and “the Kubernetes system reads the Deployment spec and starts three instances of your desired application — updating the status to match your spec. If any of those instances should fail (a status change), the Kubernetes system responds to the difference between spec and status by making a correction — in this case, starting a replacement instance.”

So the operative object in this model is not an action but a difference. Everything else in this article follows from that, including why the pipeline went green.

It is worth being precise about what the apply did and did not establish, because this is the boundary people cross without noticing.

A zero exit code from apply tells you	It does not tell you
The manifest was syntactically valid and passed admission	That the image exists or can be pulled
The object was written to the cluster’s store	That any node can satisfy the Pod’s requirements
Some controller will now be looking at it	That the new version is serving traffic, or ever will

Treating the left column as the right one is how a green pipeline coexists with an unchanged production system. A declarative apply is an acknowledgement of receipt.

Controllers are loops, and there are many of them

What acts on that difference is a control loop. The documentation borrows the term from engineering — “in robotics and automation, a control loop is a non-terminating loop that regulates the state of a system” — and uses a thermostat as the example: you set the temperature, the room has a temperature, and the device acts to close the distance.

Applied to the cluster: “in Kubernetes, controllers are control loops that watch the state of your cluster, then make or request changes where needed. Each controller tries to move the current cluster state closer to the desired state.“

Two features of how this is built matter more than the metaphor.

First, a controller usually does not do the work itself. Take the Job controller: it “does not run any Pods or containers itself. Instead, the Job controller tells the API server to create or remove Pods. Other components in the control plane act on the new information (there are new Pods to schedule and run), and eventually the work is done.”

Read that sequence again as a causal chain. One loop writes an object; a second loop notices the object and writes something else; a third acts on that. Nothing coordinates the chain end to end, and no participant knows whether the overall outcome was achieved.

Second, the multiplicity is deliberate. “As a tenet of its design, Kubernetes uses lots of controllers that each manage a particular aspect of cluster state,” and the reasoning is explicit: “it’s useful to have simple controllers rather than one, monolithic set of control loops that are interlinked. Controllers can fail, so Kubernetes is designed to allow for that.“

This is a sound design and it relocates a cost onto you. There is no transaction around “the deployment,” so there is no single place that succeeded or failed, and no single log to read. When something does not happen, the diagnostic question is which loop is not making progress, and why — which means knowing roughly which loops exist and what each one watches. The reward is that a failure in one loop does not stop the others; the price is that a stuck loop is quiet. This is the ordinary shape of partial failure in a system assembled from independent parts.

A cluster that never settles is working correctly

Here is the sentence that most changes how you operate one of these systems, and it comes from the documentation rather than from commentary:

Your cluster could be changing at any point as work happens and control loops automatically fix failures. This means that, potentially, your cluster never reaches a stable state. As long as the controllers for your cluster are running and able to make useful changes, it doesn’t matter if the overall state is stable or not.

A steady state is not the success criterion. “Converged” is not a state you wait for and then record. The system is healthy while the loops are turning and able to help.

The operational translation is a change of instrument. You are not watching for a completion event, because there is not one. You are watching two things:

• The gap between spec and status — the difference the loops exist to close.
• How long that gap has been open. This is the part teams leave out, and it is the part that carries the information. A gap is normal; a gap that has not moved in twenty minutes is an incident. Duration is what separates the two, and nothing in the platform decides the threshold for you — it is an SLO-shaped decision about how stale you are willing for reality to be.

There is a useful contrast with declarative infrastructure tools here, because they are also declarative and they behave differently. A tool driven by a state file compares declaration to record when you run it, and between runs nobody is checking — which is why configuration drift accumulates and has to be detected. See infrastructure as code and the infrastructure state file for that model. Kubernetes enforces continuously instead, so drift in the things it manages is corrected without anyone asking. The trade is that the correction is invisible unless you look, and so is its absence.

What failure looks like when nothing returns

So how does a rollout that cannot succeed announce itself? The documentation is direct about the situation: “your Deployment may get stuck trying to deploy its newest ReplicaSet without ever completing.” The listed causes are the ordinary ones — “insufficient quota, readiness probe failures, image pull errors, insufficient permissions, limit ranges, application runtime misconfiguration.”

Every one of those would be an immediate error in an imperative deployment script. Here they are conditions that persist.

The mechanism that reports it is a deadline. .spec.progressDeadlineSeconds is “an optional field that specifies the number of seconds you want to wait for your Deployment to progress before the system reports back that the Deployment has failed progressing — surfaced as a condition with type: Progressing, status: "False", and reason: ProgressDeadlineExceeded in the status of the resource.” It “defaults to 600” seconds.

Three consequences, and the third is the one that catches people.

1. Failure is a field, not an event. It appears inside the object’s status where something has to read it. A pipeline that finished ten minutes ago is not reading anything.
2. Failure has a built-in delay. At the default, roughly ten minutes pass before the condition is written at all — during which the rollout is indistinguishable from a slow one.
3. “Failed” does not mean stopped. “The Deployment controller will keep retrying the Deployment.” The condition is a report, not a terminal state. Nothing rolled back, nothing gave up, and the same image will be pulled again in a moment.

That third point is where the mental model has to change rather than be adjusted. There is no failed-and-therefore-finished. If you want a rollout to stop and revert on failure, that is a decision you implement — watch the condition and act on it. See rollback for what reverting actually requires, and canary deployment for shaping exposure while you find out.

One more property worth knowing while a rollout is in flight: the default maxUnavailable for a rolling update is 25%, so a stuck rollout can sit indefinitely with a quarter of the intended capacity missing and the rest served by the old version. Two versions running at once is the normal condition during a rollout, not an anomaly, and the persistence of that state is what a progress deadline is measuring. The compatibility obligations that come with it belong to deployment practice generally.

Since retries are unbounded and identical, whatever your workload does on startup will happen repeatedly — so startup work needs to be safe to repeat. That is idempotency applied to initialization rather than to requests, and a schema migration that runs at boot is the classic place it is forgotten. Note also that a failing pull retried in a tight loop is a load you are placing on your registry, which may answer with throttling and turn one broken rollout into a cluster-wide slowdown; the general remedy is backoff, and the platform’s own retry behaviour is not yours to tune.

Scheduling is one decision, made once

Creating a Pod object does not place it. “A scheduler watches for newly created Pods that have no Node assigned,” and for each one “becomes responsible for finding the best Node for that Pod to run on.”

The decision is two steps. “The filtering step finds the set of Nodes where it’s feasible to schedule the Pod” — nodes meeting the Pod’s declared requirements, which the documentation calls feasible nodes. Then “in the scoring step, the scheduler ranks the remaining nodes to choose the most suitable Pod placement,” and the highest score wins. Ties are broken arbitrarily: “if there is more than one node with equal scores, kube-scheduler selects one of these at random.” Finally the scheduler “notifies the API server about this decision in a process called binding.”

Two things follow, and both are common sources of confusion.

An unplaceable Pod waits silently and indefinitely. “If none of the nodes are suitable, the pod remains unscheduled until the scheduler is able to place it.” There is no timeout in that sentence and no error. A Pod whose requirements no node can meet — because the cluster is full, or because it asked for a node type you do not have — simply sits, and the Deployment above it shows an unclosed gap rather than a failure. This is also where cluster-level resource exhaustion shows up: not as an out-of-memory crash but as things quietly not starting.

Binding is not revisited. The scheduler’s job is to place Pods that have no node. Once a Pod is bound, that Pod stays where it is for its lifetime. Add nodes to a full cluster and the pending Pods get placed, but the existing ones do not move; the cluster is now unbalanced and will stay that way until those Pods are replaced for some other reason. Placement is continuously decided for new Pods and never revised for old ones — the loop watches for unassigned Pods, not for suboptimal ones.

The practical habit that falls out of this: when a workload must be spread or kept apart, express the requirement in the Pod’s spec so that filtering enforces it, rather than relying on the scheduler’s scoring to have arranged things sensibly. Scoring optimizes at the moment of placement, against the cluster as it was then.

Choosing a workload object is choosing what identity means

You rarely declare Pods directly. “Managing individual Pods would be a lot of effort,” so you create a higher-level object and “the Kubernetes control plane automatically manages Pod objects on your behalf.” Which object you pick is not a matter of taste; each one encodes a different answer to whether one replica is distinguishable from another.

Object	What it assumes	Use when
Deployment	“Any Pod in the Deployment is interchangeable and can be replaced if needed”	A stateless workload where replicas are fungible
StatefulSet	“The Pods rely on having a distinct identity. This is different from a Deployment where the Pods are expected to be interchangeable”	Each replica must keep its own storage — “if one of the Pods in the StatefulSet fails, Kubernetes makes a replacement Pod that is connected to the same PersistentVolume”
DaemonSet	The work belongs to a node, not to the application	“Pods that provide facilities that are local to a specific node” — log shippers, storage drivers, node agents
Job / CronJob	The work ends	“Tasks that run to completion and then stop” — a Job is one-off, a CronJob repeats on a schedule

The expensive mistake is running something that has identity under an object that assumes it does not. A Deployment will replace a Pod with an equivalent one wherever it fits, which is precisely the right behaviour for a stateless service and precisely wrong for a replica that owns a volume or holds a position in a cluster of its own.

And the Deployment case has a subtlety the documentation names in passing: it manages Pods “indirectly” through a ReplicaSet. That intermediate object is why a rollout can be described, paused, and reverted — there is an old set and a new set, both real objects. It is also why a stuck rollout leaves two ReplicaSets in place, which is worth recognizing rather than tidying away.

The general point is that Pods are not durable things to reason about. “Pods are ephemeral resources (you should not expect that an individual Pod is reliable and durable),” and they are “created and destroyed to match the desired state of your cluster.” Anything that must outlive a Pod — data, identity, a record of what happened — has to live somewhere the Pod’s lifecycle does not reach. What a Pod is and what it shares is covered under container and container image; here it is enough that the Pod is the unit that gets scheduled, and the unit that gets thrown away.

The joints between objects are label selectors

If controllers do not call each other, how does a Deployment know which Pods are its own, or a Service know where to send traffic? Through labels, and the documentation is unambiguous about the weight this carries: “the label selector is the core grouping primitive in Kubernetes.“

Labels are “key/value pairs that are attached to objects,” intended to “specify identifying attributes of objects that are meaningful and relevant to users, but do not directly imply semantics to the core system.” They exist so that users can “map their own organizational structures onto system objects in a loosely coupled fashion.” That looseness is the design goal and the hazard in the same sentence.

Three properties are worth memorizing because each one produces a distinct failure.

• Labels do not identify anything uniquely. “Unlike names and UIDs, labels do not provide uniqueness.” A selector matches whatever happens to carry the labels — including objects you did not have in mind.
• Selectors combine with AND only. Multiple requirements are comma-separated and “the comma separator acts as a logical AND” — and explicitly, “for both equality-based and set-based conditions there is no logical OR operator.” A grouping you cannot express as a conjunction has to be expressed as a label you add on purpose.
• An empty selector is not a neutral value. “The semantics of empty or non-specified selectors are dependent on the context” — in some places it means everything and in others nothing. It is not a safe default, and an omission is not a no-op.

The everyday version of the first property is the one that eats an afternoon. Change the label on your Pods, or mistype the selector on your Service, and the Service now matches nothing. It is a valid object. It has an address. Connections to it fail, and there is no error message anywhere naming the mismatch, because an empty match is a legitimate result of a correct query. The symptom of a selector typo is silence, not a validation failure — which is the direct consequence of “loosely coupled” being a design goal.

Overlap is the other side of it, and here the documentation states the consequence outright: “for some API types, such as ReplicaSets, the label selectors of two instances must not overlap within a namespace, or the controller can see that as conflicting instructions and fail to determine how many replicas should be present.” Two owners disagreeing about the same Pods is not a detected conflict; it is two loops pulling in different directions, which looks like replicas appearing and disappearing for no reason.

Labels are also how controllers stay out of each other’s way in the normal case. Deployments and Jobs both create Pods, and “the Job controller does not delete the Pods that your Deployment created, because there is information (labels) the controllers can use to tell those Pods apart.” The isolation between your workloads rests on a naming convention.

Which makes label conventions a platform responsibility rather than a matter of local style. If every team invents its own keys, then selectors across teams cannot be written reliably, policies cannot target workloads, and cost attribution has no grouping to work with. This is exactly the class of cross-cutting standard that platform engineering exists to own, and it is cheap to establish early and expensive to retrofit once thousands of objects carry ad-hoc keys.

Services exist because Pods do not last

Put the last two sections together and a problem appears. Pods are replaced routinely, each has its own address, and “for a given Deployment in your cluster, the set of Pods running in one moment in time could be different from the set of Pods running that application a moment later.” So how does a caller find a callee?

A Service. “Each Service object defines a logical set of endpoints (usually these endpoints are Pods) along with a policy about how to make those pods accessible,” and “the set of Pods targeted by a Service is usually determined by a selector that you define.” Behind it, “Kubernetes updates the EndpointSlices for a Service whenever the set of Pods in a Service changes” — another loop, watching membership.

The design intent is stated plainly and it is a compatibility promise: “a key aim of Services in Kubernetes is that you don’t need to modify your existing application to use an unfamiliar service discovery mechanism.” A containerized application that resolves a hostname and opens a connection keeps working. Discovery is provided by the platform rather than adopted by the code, which is why moving an existing service onto Kubernetes does not usually require touching how it finds its dependencies.

The part to hold on to here is that the stable name is decoupled from the unstable set by a selector. The name is reliable; the membership behind it is a query result that changes continuously. How that traffic is actually routed, and what happens to connections during the change, is a networking matter beyond this article’s scope.

These namespaces are not the kernel’s namespaces

One term deserves its own section because the collision is genuinely confusing and both meanings show up in the same conversation.

A Linux namespace is a kernel isolation mechanism: it changes what view of a system resource a process has. A Kubernetes namespace is an organizational boundary inside a cluster’s API: “namespaces provide a mechanism for isolating groups of resources within a single cluster. Names of resources need to be unique within a namespace, but not across namespaces.” Same word, different layer, unrelated mechanism.

Two clarifications keep the Kubernetes one in proportion.

• It does not cover everything. “Namespace-based scoping is applicable only for namespaced objects (e.g. Deployments, Services, etc.) and not for cluster-wide objects (e.g. StorageClass, Nodes, PersistentVolumes, etc.).” A boundary with exceptions is not a boundary you can lean on for separation — relevant to both workload isolation and tenant isolation, and the reason multi-tenancy on a shared cluster needs more than namespaces.
• It is not the tool for every kind of separation. “It is not necessary to use multiple namespaces to separate slightly different resources, such as different versions of the same software: use labels to distinguish resources within the same namespace.” Namespaces are for names and quota — they are “a way to divide cluster resources between multiple users (via resource quota).” Variants of one thing are a labelling problem.

What the model buys, and what it charges

The gain is real and easy to state. Declare intent once and the platform keeps working toward it — a failed node’s Pods are recreated elsewhere, a crashed process is restarted, a deleted object is put back, and none of it requires anyone to be paged or to run anything. Self-healing is not a feature bolted on; it is what a non-terminating loop does by default.

The charges are the module’s real subject, and there are four.

1. You have to monitor gaps and their age. Nothing reports completion, so the absence of a report is not news. A dashboard built on deploy events will show a successful deployment that never happened.
2. You have to decide what identity means for every workload before you choose an object, because the object enforces an answer either way — and getting it wrong is silent until a replica loses its volume.
3. Your coupling is a convention. Selectors bind objects together, and a convention has no compiler. Label discipline is infrastructure.
4. Every abstraction is a loop that can be behind. “Applied” means accepted, never done. Any question of the form “is it live?” has to be answered by reading status, not by trusting an earlier command — which also means the state of the cluster and the record of who asked for what are different sources, so reconstructing an incident needs both: see audit log.

Which brings the opening scenario to a close. The pipeline was green because the API server accepted a description; the rollout was stuck on an image that could not be pulled; the Deployment controller had written ProgressDeadlineExceeded into a status field about half an hour earlier and was still retrying; and nothing was reading that field. Every component behaved exactly as documented. The gap was in the expectation that somebody, somewhere, was running the manifest.

References, all read September 2026: Kubernetes: Objects In Kubernetes; Controllers; Workload Management; Deployments; Kubernetes Scheduler; Labels and Selectors; Service; Namespaces. API field names and default values are versioned and may change.