Ephemeral by Default: Networking, Storage, and Staying Alive on Kubernetes

A team moves a reporting service onto a cluster. It works. Three weeks later someone deletes a namespace that was supposed to be idle, and the database volume goes with it — not the Kubernetes object, the actual disk in the cloud account. Nobody chose that. The storage class had a default nobody read.

(The scenario is illustrative.) It is the same shape as three other surprises that arrive in the first months: a service that is unreachable because nothing exposed it, two replicas quietly corrupting one volume, and a rollout that drops requests because the platform stopped sending traffic before the application stopped accepting it.

All four come from one property. On a container platform the default is that things disappear. A container is replaceable, a Pod is replaced routinely, and when it goes it takes its filesystem and its IP address with it. So connection, persistence, resource share, and survival do not come with the platform — each has to be declared. And the second half of this article is about what a declaration actually guarantees, which is consistently less than it appears.

Quotations are from the Kubernetes documentation as read in September 2026. Field names, defaults, and feature maturity are versioned.

Exposure is a layered choice, not a switch

A Service gives a stable name over a changing set of Pods, and its controller “continuously scans for Pods that match its selector, and then makes any necessary updates to the set of EndpointSlices” — another loop, joined to the workload by a label selector. What decides who can reach it is the type, and the four types are not alternatives: “the type field in the Service API is designed as nested functionality — each level adds to the previous.“

Type	Reachable from	What it costs you
ClusterIP (default)	Inside the cluster only — “choosing this value makes the Service only reachable from within the cluster”	Nothing. External access needs “an Ingress or a Gateway”
NodePort	Every node’s IP at a fixed port. It “sets up a cluster IP address, the same as if you had requested a Service of type: ClusterIP”	A port on every node, and callers that need to know node addresses
LoadBalancer	Outside, through a provisioned balancer	A real cloud resource with a bill. Note that “Kubernetes does not directly offer a load balancing component; you must provide one”
ExternalName	N/A — a DNS alias to an outside host	Nothing, and it does nothing: “no proxying of any kind is set up“

Two consequences worth acting on. A LoadBalancer per service is a cost decision disguised as a field. The documentation describes the implementation — Kubernetes “typically starts off by making the changes that are equivalent to you requesting a Service of type: NodePort,” and “the cloud-controller-manager component then configures the external load balancer to forward traffic to that assigned node port” — so ten LoadBalancer services are ten cloud balancers. Consolidating HTTP behind one entry point is what Ingress and Gateway exist for, and it is the same consolidation argument as an API gateway.

And provisioning is asynchronous: “the actual creation of the load balancer happens asynchronously,” with the result published in the Service’s status. So a Service that exists is not a Service that is reachable, and a pipeline that creates one and immediately smoke-tests the external address will fail intermittently for reasons that have nothing to do with the application. The fix is to wait on status rather than on object creation — the same discipline as everywhere else in a system built on desired state.

One more type exists for the case where you do not want a virtual IP at all. A headless Service — cluster IP explicitly set to “None” — means “a cluster IP is not allocated, kube-proxy does not handle these Services, and there is no load balancing or proxying done by the platform“; instead it reports “the endpoint IP addresses of the individual pods via internal DNS records.” That is what you want when the client needs to address a specific replica, which is exactly the case for the stateful workloads in the next section.

Storage is a separate system with its own lifecycle

The documentation opens with the framing that explains the whole API: “managing storage is a distinct problem from managing compute instances.” Two objects follow from that separation, and the split is the point.

• A PersistentVolume is “a piece of storage in the cluster,” and critically it has “a lifecycle independent of any individual Pod that uses the PV.”
• A PersistentVolumeClaim is “a request for storage by a user.” The symmetry the docs draw is the useful mnemonic: “Pods consume node resources and PVCs consume PV resources.“

So your workload asks for storage with a claim and never names a disk, which is what lets the same manifest run in three environments. Where the volume comes from is either static (an administrator created PVs in advance) or dynamic, and dynamic provisioning “is based on StorageClasses: the PVC must request a storage class and the administrator must have created and configured that class.” Note that this is a different layer from mounting a directory into a container at runtime — that is a container-runtime concern, while these are cluster resources with their own objects and permissions.

The default that deletes your data

Here is the opening scenario, explained. A PV’s reclaim policy “tells the cluster what to do with the volume after it has been released of its claim,” and the options are Retain, Recycle, or Delete. Under Delete, “deletion removes both the PersistentVolume object from Kubernetes, as well as the associated storage asset in the external infrastructure.”

And the default: “volumes that were dynamically provisioned inherit the reclaim policy of their StorageClass, which defaults to Delete.“

Put those together. Dynamic provisioning is the convenient path, the convenient path defaults to Delete, and Delete destroys the real disk. So deleting a PVC — or a namespace containing one — can destroy production data, and the manifest that did it contains no mention of deletion. The alternative is Retain, under which “when the PersistentVolumeClaim is deleted, the PersistentVolume still exists and the volume is considered ‘released’,” with the data intact and manual steps required to reuse it.

Two things to do about it, and the first takes a minute. Read the reclaim policy of the storage classes your cluster actually offers, rather than assuming. Then decide it per workload: a cache volume genuinely should be deleted with its claim, and a database volume should not. Whatever you conclude, a reclaim policy is not a backup — see audit log for reconstructing who asked for the deletion afterwards, which is a different question from recovering the bytes.

Access modes match; they do not enforce

This is the most consequential misreading in Kubernetes storage, and the documentation is explicit about both halves of it.

First the scope. ReadWriteOnce means “the volume can be mounted as read-write by a single node” — and then the sentence people miss: “ReadWriteOnce access mode still can allow multiple pods to access (read from or write to) that volume when the pods are running on the same node.” So RWO is not “one Pod.” Two replicas scheduled onto the same node can both write to a volume that was labelled RWO, and a filesystem not designed for that will corrupt quietly. The mode that means one Pod is ReadWriteOncePod, “Stable since Kubernetes v1.29,” and the project’s own example annotation recommends it: “for production use, the Kubernetes project recommends using the ReadWriteOncePod access mode instead.”

Then the enforcement, or its absence: “Kubernetes uses volume access modes to match PersistentVolumeClaims and PersistentVolumes… Volume access modes do not enforce write protection once the storage has been mounted. Even if the access modes are specified as ReadWriteOnce, ReadOnlyMany, or ReadWriteMany, they don’t set any constraints on the volume. For example, even if a PersistentVolume is created as ReadOnlyMany, it is no guarantee that it will be read-only.” The exception is ReadWriteOncePod, where “the volume is constrained and can be mounted on only a single Pod.”

Read as a class of fact, this is the article’s thesis in miniature: a declared property can be matching metadata rather than a constraint, and the two are indistinguishable in the manifest. Also note that “a volume can only be mounted using one access mode at a time, even if it supports many,” so a volume’s capability list is not a description of how it is currently being used.

Identity, storage, and a DNS trap

Where replicas are not interchangeable, the pieces connect. A StatefulSet gives each Pod its own claim — “for each VolumeClaimTemplate entry defined in a StatefulSet, each Pod receives one PersistentVolumeClaim” — and a stable name, $(statefulset name)-$(ordinal), with a headless Service to “control the domain of its Pods.”

The deletion behaviour here is the opposite of the earlier default, which is worth knowing precisely because it is inconsistent with it: “the PersistentVolumes associated with the Pods’ PersistentVolume Claims are not deleted when the Pods, or StatefulSet are deleted. This must be done manually.” So scaling down leaves volumes behind — safe for the data, and a quiet cost line.

And one operational trap that looks like a bug in your application. Because DNS caches failures, “you may not be able to look up the DNS name for a newly-run Pod immediately,” because “negative caching (normal in DNS) means that the results of previous failed lookups are remembered and reused, even after the Pod is running, for at least a few seconds.” A cluster member that starts, resolves its peers, gets nothing, and gives up is hitting this. The documented options are to “query the Kubernetes API directly (for example, using a watch) rather than relying on DNS lookups,” or to reduce the DNS cache time — and the application-side answer is to retry rather than exit, which is backoff applied to startup.

Requests and limits are two different mechanisms

Every container can declare what it needs and what it may not exceed, and the two numbers act at different times through different machinery. Getting this wrong produces either idle clusters or midnight restarts.

A request is a scheduling claim, not a measurement. The scheduler “ensures that, for each resource type, the sum of the resource requests of the scheduled containers is less than the capacity of the node” — and the next sentence is the one that explains a cluster that looks empty and refuses work: “although actual memory or CPU resource usage on nodes is very low, the scheduler still refuses to place a Pod on a node if the capacity check fails.” The reason is deliberate: it “protects against a resource shortage on a node when resource usage later increases, for example, during a daily peak.”

So requests reserve. Inflated requests buy nothing and cost capacity, which is how a cluster ends up at 15% utilisation and unable to schedule anything. See dynamic resource allocation for the elastic version of this problem.

Exceeding CPU delays you; exceeding memory kills you

This asymmetry is the single most useful thing to know about limits, and it follows from how the kernel enforces each.

	CPU	Memory
Request does	Sets a weighting — “workloads with larger CPU requests are allocated more CPU time than workloads with small requests”	Mainly scheduling. Exceed it on a pressured node and “it is likely that the Pod… will be evicted“
Limit does	“A hard ceiling”; when exceeded “the kernel waits before allowing that cgroup to resume execution”	“The Linux kernel out-of-memory subsystem activates and, typically, intervenes by stopping one of the processes“
Consequence	Slower. “Container runtimes don’t terminate Pods or containers for excessive CPU usage.“	Dead. If the killed process is PID 1 “and the container is marked as restartable, Kubernetes restarts the container”

Two practical rules come straight out of that table. A CPU limit set too low produces latency, not errors — and latency with no error log, which is why a service that is mysteriously slow under load is worth checking against its CPU limit before its code. (This is kernel-level delay of a cgroup, a different mechanism from the request-rate throttling an API applies to a caller, though the word is used for both.) A memory limit set too low produces restarts, and a restart loop under load looks like an application bug.

The gap between request and limit is where noisy neighbours live. Setting requests low and limits high lets a container burst into capacity the node did not reserve for it, which works until several do it at once — see noisy neighbor and resource exhaustion. Where isolation between tenants or workloads has to be guaranteed rather than hoped for, resource declarations are one layer of it and not the whole answer: see workload isolation, tenant isolation, and multi-tenancy.

One unit trap deserves its own line because it is silent and absurd: “‘M’ means megabytes, while ‘m’ means millibytes. If you request 400m of memory, this is a request for 0.4 bytes.” For CPU the milli form is the safer habit — 1 CPU unit “is equivalent to 1 physical CPU core,” 0.1 equals 100m, and precision “finer than 1m” is rejected.

Three probes, three different decisions

The kubelet “can optionally perform and react to three kinds of probes on running containers, each serving a different purpose.” Conflating them is the most common configuration error in this area, and the way to keep them apart is to ask what each one causes.

Probe	Question	Failure causes	Runs
Startup	Has it finished starting?	The kubelet “kills the container”	At startup only
Liveness	Is it stuck?	Restart of the container	Periodically
Readiness	Should it receive traffic?	Removal from Service endpoints — no restart	“During its whole lifecycle”

The readiness mechanism is worth stating precisely because it is the link back to the networking section: “if the readiness probe returns a failed state, the EndpointSlice controller removes the Pod’s IP address from the EndpointSlices of all Services that match the Pod.” Traffic stops; the process keeps running. That is the right response to a temporary condition — the documentation notes readiness is useful “when recovering from temporary faults or overloads,” which is backpressure expressed through the platform’s routing rather than through the protocol.

Liveness carries a warning the documentation states unusually strongly: probes “must be configured carefully to ensure that they truly indicate unrecoverable application failure, for example a deadlock,” because “incorrect implementation of liveness probes can lead to cascading failures. This results in restarting of container under high load; failed client requests as your application became less scalable; and increased workload on remaining pods due to some failed pods.”

Follow that sequence, because it is a genuine outage pattern rather than a caution. Load rises, response times rise, the liveness probe times out, the container is restarted, its share of traffic moves to the remaining Pods, their response times rise, and their probes begin to fail. A liveness probe that measures load rather than deadlock converts a slow service into an unavailable one. So a liveness endpoint should check that the process can still make progress, not that a dependency is healthy or that the service is fast — those belong to readiness and to an SLO respectively. The general shape is familiar from partial failure: a local recovery mechanism that fires on the wrong signal amplifies the fault.

Startup probes exist to stop the two above from interfering with slow initialization. When one is configured, “Kubernetes does not execute liveness or readiness probes until the startup probe succeeds,” and the documented decision rule is arithmetic: if the container usually takes longer than \(initialDelaySeconds + failureThreshold \times periodSeconds\) to start, add a startup probe checking the same endpoint as liveness. The default period is 10 seconds.

Shutting down is a protocol, and your code is in it

Pods are deleted constantly — every rollout, every scale-down, every node drain. The platform’s aim is stated plainly: to let processes “gracefully terminate when they are no longer needed (rather than being abruptly stopped with a KILL signal and having no chance to clean up).” What it provides is a signal and a deadline.

The sequence: the kubelet stops containers “by first sending a TERM (aka. SIGTERM) signal, with a grace period timeout, to the main process in each container,” and then “once the grace period has expired, the KILL signal is sent to any remaining processes, and the Pod is then deleted from the API Server.” In parallel, “when the Pod is deleted, the corresponding endpoint in the EndpointSlice will update its conditions: the endpoint ready condition will be set to false, so load balancers will not use the Pod for regular traffic.”

Which means graceful shutdown is a contract with two parties, and the platform only holds up its end. Four things the application has to do, and the first is where most services fail:

1. Handle SIGTERM. A process that ignores it gets the full grace period of nothing followed by a KILL. Note also that the signal may not be TERM: “many container runtimes respect the STOPSIGNAL value defined in the container image and, if different, send the container image configured STOPSIGNAL instead.”
2. Stop accepting new work, then finish what is in flight. Two separate actions — the second is what prevents dropped requests.
3. Fit inside the grace period, or set one that fits the work. A job that needs 90 seconds to flush under a 30-second grace period is killed mid-flush every time.
4. Be safe to interrupt anyway. KILL always remains possible — a node can fail outright — so anything that must not be done twice needs idempotency rather than a clean exit.

Two details in the mechanism that break plausible assumptions. Containers are not stopped in a defined order: the requests “are processed by the container runtime asynchronously. There is no guarantee to the order of processing for these requests.” So a sidecar that ships logs or proxies connections may exit before the application it serves, and a shutdown sequence that depends on ordering will work in testing and fail under load. And the clock can restart: “if the kubelet or the container runtime’s management service is restarted while waiting for processes to terminate, the cluster retries from the start including the full original grace period.“

There is also a race worth designing against. Endpoint removal and SIGTERM are concurrent, and load balancers upstream of the cluster may take longer still to stop sending traffic. A process that exits the instant it receives SIGTERM can therefore refuse requests that are already on their way. The usual remedy is to fail readiness first, keep serving briefly, and only then begin draining — which is why the readiness probe and the shutdown path are one design rather than two.

What this layer adds to your job

The gain is that a workload becomes portable: it asks for storage without naming a disk, is reachable by a name it does not own, and declares what it needs without knowing which machine will provide it. That is real, and it is why the same manifests run in three environments.

The costs are four, and every one of them is a declaration that has to be right.

• Storage lifecycle is now yours to state. The convenient default deletes the disk with the claim. Deciding per workload is a five-minute task that has no obvious owner until it has cost something.
• Some declarations match rather than enforce. Access modes are the clear case, and reading them as guarantees is how two writers end up on one volume.
• Resource numbers have two distinct effects each, and the CPU and memory failure modes are nothing alike — silent slowness versus a restart loop.
• Your application is part of the platform’s lifecycle protocol. Probes and signal handling are application code that the platform depends on, so “it works when we run it” is not the same as “it behaves correctly when the platform starts and stops it.”

Which closes the opening scenario. The volume was deleted because a dynamically provisioned PV inherits a storage class whose reclaim policy defaults to Delete; the unreachable service had the default ClusterIP type; the corrupted volume was ReadWriteOnce with two Pods on one node; and the dropped requests came from a process exiting on SIGTERM while traffic was still arriving. Every component behaved as documented. What was missing in each case was a declaration, or an accurate reading of one.

References, all read September 2026: Kubernetes: Service; Persistent Volumes; StatefulSets; Resource Management for Pods and Containers; Liveness, Readiness, and Startup Probes; Pod Lifecycle. API field names, default values, and feature maturity are versioned and do change.