Operating Shared Kubernetes Clusters

One cluster serves several teams. That arrangement shares nodes, a control plane, networking, and an upgrade calendar, and every one of those is a place where one team’s action can change another team’s day. A batch job without resource limits crowds out an API. A manifest deployed with a deprecated API version works until the cluster is upgraded. A policy that rejects unsafe workloads blocks an urgent fix because nobody has an exception path.

This post is about operating that arrangement on purpose. It covers how to isolate tenants and bound their resource use, how to use admission policy so the rules are enforced by the API server rather than by memory, how to reason about version compatibility, how to stage an upgrade so that a bad one affects few workloads, and what rollback can and cannot do. The scenarios are illustrative. They use made-up teams named team-a and team-b and assumed numbers, and none describes a real organization.

What this post assumes

It builds on Kubernetes networking, storage, and stateful workloads. From that post it assumes you know how Services select ready Pods through EndpointSlices, that cluster DNS depends on network paths a default-deny policy can break, that a PersistentVolumeClaim and its reclaim policy decide whether data survives deletion, and that a StatefulSet gives each Pod an identity and a disk but does not replicate data. Those facts reappear in the sections on isolation and node maintenance, and each reappearance explains the part it needs. General operating-systems knowledge, such as CPU scheduling and out-of-memory behavior, is assumed.

A note on evidence. No Kubernetes cluster was available while this post was written. The kubectl commands and manifests were not applied, and the manifests were checked only for YAML syntax. CEL expressions in the admission policy example were not evaluated. Small Python scripts reproduce documented rules and arithmetic: the QoS class rule, quota and surge headroom, the PodDisruptionBudget allowance, and the version skew rule. The Python blocks are short calculation aids that make a documented rule or an assumed number concrete. They are excerpts: the output shown came from fuller local versions that also print headers and extra rows. Their output is shown where it is used. Behavior described as documented comes from the Kubernetes documentation in the references, and inferences are labeled as such.

What a tenant is, and how much isolation is enough

A tenant is a group served by a shared system. The glossary entry on multi-tenancy describes the two patterns the Kubernetes documentation names: multi-team tenancy, where teams inside one organization share a cluster and have some trust in each other, and multi-customer tenancy, where a vendor runs workloads for customers who may not trust each other. The Kubernetes multi-tenancy documentation presents soft and hard tenancy as a spectrum rather than two categories. The difference is how much you rely on tenants not attacking, or accidentally harming, each other.

Pattern	What is shared	What it relies on	Cost and complexity
Namespace per tenant	Nodes, control plane, cluster-scoped objects	RBAC, quotas, network policies, admission policy configured correctly	Lowest; suits teams with some mutual trust
Virtual control plane per tenant	Nodes (usually), underlying cluster	An added layer that gives each tenant its own API server	Higher; more components to operate
Cluster per tenant	Nothing at the cluster level	Separate clusters and their own upgrade work	Highest; every cluster needs its own upgrades, policies, and monitoring

The pattern descriptions follow the Kubernetes multi-tenancy page. The table intentionally has no column that says which is best. The right row depends on how much tenants trust each other, what a failure costs, and how many clusters the team can operate. Cluster per tenant removes shared fate but multiplies the upgrade and policy work discussed later. A shared cluster reduces that work and ties tenants to one upgrade schedule and one control plane. This post addresses the namespace-per-tenant pattern because it is the common starting point, and it points out where its limits begin.

Isolation is not one property. Access, performance, cost, and blast radius can each be separated independently, and a design can be strong on one and weak on another. The glossary entries on tenant isolation, noisy neighbors, and blast radius cover the access, performance, and failure-scope dimensions in short form. The rest of this section applies them to Kubernetes primitives.

Isolation and quotas

Namespaces and RBAC: who may touch what

A namespace is the primary isolation mechanism for API objects: names are scoped to it, and security policy such as RBAC and network policy can be scoped to it. The multi-tenancy page recommends a namespace per tenant and RBAC to restrict users and service accounts to their namespace. The role-based access control entry explains how role assignments express who may do what, and least privilege gives the principle for sizing the grant: only the operations and the scope the work needs.

Namespaces do not partition everything. We observe from the object model, and the glossary entry on blast radius states it directly, that cluster-scoped objects such as CustomResourceDefinitions, StorageClasses, and admission webhook configurations are shared by all tenants. A tenant allowed to create one of those can change behavior for everyone. Tenant roles should therefore be namespaced Roles or ClusterRoles bound with a RoleBinding in the tenant namespace, and they should not include permission to create cluster-scoped resources.

Network isolation

By default, Pods in a cluster can communicate with each other regardless of namespace. The network policy entry and the Kubernetes documentation describe the mechanism: a policy selects Pods and allows specified connections, policies are additive, and enforcement requires a network plugin that implements them. A default-deny policy per tenant namespace, plus explicit allows, is the usual starting point. As the previous post covered, a default-deny egress policy also blocks DNS, so the first allow rule should permit the cluster DNS service. A test that crosses the tenant boundary, from a Pod in one namespace to a Pod in another, is the way to know the plugin enforces what the API server accepted.

Compute: requests, limits, and what the kernel does with them

Resource isolation starts with what each container declares. The documentation on managing container resources separates two numbers. A request is used by the scheduler to place the Pod, and the kubelet reserves at least that amount. A limit is enforced while the container runs. The enforcement differs by resource. CPU limits are enforced by throttling, so a container that wants more CPU than its limit gets less processor time. Memory limits are enforced reactively: if a container exceeds its limit when the node is under memory pressure, the kernel may kill it, and the documentation notes that a container can temporarily exceed its memory limit before that happens. If you set a limit but no request, Kubernetes uses the limit as the request.

The Pod quality-of-service class follows from those fields. A Pod is Guaranteed if every container has memory and CPU requests equal to its limits. It is BestEffort if no container sets any CPU or memory request or limit, and Burstable otherwise. Under node resource pressure, BestEffort Pods are evicted first, then Burstable, with Guaranteed last. The next block is a model of the QoS rule, run with four sample specifications.

def qos(containers):
    # each container: dict with optional cpu_req, cpu_lim, mem_req, mem_lim
    if all(c.get(k) is None for c in containers
           for k in ("cpu_req", "cpu_lim", "mem_req", "mem_lim")):
        return "BestEffort"
    g = all(c.get("cpu_req") and c.get("cpu_req") == c.get("cpu_lim") and
            c.get("mem_req") and c.get("mem_req") == c.get("mem_lim")
            for c in containers)
    return "Guaranteed" if g else "Burstable"

For no resources set, equal requests and limits, a CPU limit above the request, and requests with no limits, the script printed these classes in order.

BestEffort
Guaranteed
Burstable
Burstable

This model covers only the classification rule from the documentation. It does not model Pod-level resource settings, which the documentation also mentions, and it says nothing about how much capacity a node has. Its purpose is to make the consequence of leaving fields blank concrete: a workload without requests is the first candidate for eviction, and the scheduler counts it as needing nothing when it places it.

ResourceQuota: a per-namespace budget

A ResourceQuota constrains aggregate consumption in a namespace. According to the documentation, it can limit the sum of CPU and memory requests and limits (requests.cpu, limits.memory, and so on), storage requested by PersistentVolumeClaims (requests.storage), and counts of objects such as Pods and PVCs. Three behaviors matter in operation.

• Enforcement is at admission. A request that would exceed the quota is rejected with HTTP 403 Forbidden when the object is created. Existing objects are not affected, and quotas are not retroactive. The documentation also describes quota usage as eventually consistent, so usage shown in the quota’s status can lag behind recent creations.
• Requests become mandatory. When a quota covers CPU or memory, every new Pod must specify the corresponding requests or limits, or it is rejected. A LimitRange can inject defaults so that tenants are not rejected for omitting them.
• Scopes narrow what is counted. Scopes such as BestEffort, NotBestEffort, Terminating, NonTerminating, and PriorityClass let separate quotas apply to different kinds of Pods.

A LimitRange constrains individual objects instead of the namespace total. It can set minimum and maximum values, a maximum ratio of limit to request, and default requests and limits applied when a container omits them. The documentation makes two cautions explicit. LimitRange constraints apply only when objects are created or updated, and modifying a LimitRange does not change running Pods. And defaults are not checked against each other: if the default limit is 500m and a Pod sets a request of 700m with no limit, the Pod is rejected because its request exceeds the injected default limit. The remedy is to set both request and limit explicitly. When several LimitRanges exist in a namespace, which defaults apply is not deterministic, so keep one.

A worked quota example, and the surge trap

Quota arithmetic interacts with rollouts. Suppose, as an assumption for this example, that team-a has a quota of 10 CPU of requests, and its Deployment has 18 replicas that each request 500m. Steady-state usage is 9 CPU, which fits. The Deployment documentation gives the default rolling update settings: maxSurge and maxUnavailable both default to 25%, maxSurge rounds up, and maxUnavailable rounds down. For 18 replicas that is up to 5 extra Pods during an update and up to 4 unavailable. The script below computes the headroom.

import math
quota_cpu_m, pod_req_m, replicas = 10_000, 500, 18
used = replicas * pod_req_m
max_surge = math.ceil(replicas * 0.25)        # percentage rounds up
max_unavail = math.floor(replicas * 0.25)     # percentage rounds down
fit = (quota_cpu_m - used) // pod_req_m
print(used, max_surge, max_unavail, fit, max(0, max_surge - fit))

The full script printed this output.

steady state requests.cpu used (m): 9000 of 10000
maxSurge pods: 5  maxUnavailable pods: 4
headroom (m): 1000 -> surge pods that fit under quota: 2 of 5
surge pods that would get 403 Forbidden: 3
request > injected default limit -> rejected: True

Under these assumptions, 3 of the 5 surge Pods would be rejected with a quota error. The Deployment documentation says that insufficient quota can stall a rollout and sets a ReplicaFailure condition on the Deployment, with a default progressDeadlineSeconds of 600. Whether this particular rollout stalls or completes more slowly depends on how the controller combines surge and unavailability, which the script does not model. The lesson is operational rather than predictive. A quota sized exactly to steady-state usage leaves no room for rollouts, node-drain replacement, or autoscaling, so quota headroom should be a stated policy, and a quota error during a rollout should be among the first things checked.

What quotas do not cover

A quota bounds the declared request sum in a namespace. It does not make the node’s shared resources private. The multi-tenancy page and the glossary entry on noisy neighbors note that quotas do not protect against every kind of sharing, such as network bandwidth, and that when requests are lower than limits, some impact between workloads remains possible. Storage throughput, local disk, connection pools, and the shared API server are other contended resources.

The API server is a shared resource that namespaces do not partition. API Priority and Fairness, stable since Kubernetes 1.29 according to the documentation, classifies requests with FlowSchemas into priority levels with separate concurrency limits, and uses fair queuing so one flood of requests does not starve other flows. It is the mechanism that keeps a misbehaving controller or script in one tenant from taking all API capacity. The documentation describes it as an alternative to the older max-in-flight flags. Its configuration is cluster-level, and a platform team, not a tenant, owns it.

Priority is another place where tenants interact. A PriorityClass gives Pods a numeric priority, and when a Pod cannot be scheduled the scheduler can preempt lower-priority Pods to make room. The documentation warns that in a multi-tenant cluster a user could create high-priority Pods that evict others, and it recommends using ResourceQuota to limit consumption by priority class. A tenant that can set any priorityClassName can effectively decide whose Pods are evicted.

Admission policies: rules enforced where objects are created

Isolation settings only work if they are present. A tenant namespace that was created without a quota, or a Deployment that omits limits, silently weakens the design. Admission control is the point in the API server where such requirements can be enforced for every request.

Where admission sits

According to the documentation, admission controllers intercept requests to create, modify, or delete resources after authentication and authorization and before the object is persisted. They do not act on reads. There are two kinds: mutating controllers, which can change the object, and validating controllers, which can only accept or reject. Mutating controllers run first, then validating controllers, and if any controller rejects the request, the request fails. A mutating webhook that changes the object causes earlier webhooks to be invoked again to validate the result. Several controllers are enabled by default and are part of what makes the isolation settings work, including LimitRanger, ResourceQuota, NamespaceLifecycle, and PodSecurity. The request-time quota rejection described above is performed by admission.

Four extension points let a cluster add its own rules: mutating and validating webhooks, which call an external HTTPS service, and mutating and validating admission policies, which are declarative and run inside the API server. The glossary entry on policy as code describes the practice that applies here: rules expressed as reviewed, versioned, executable checks, with attention to the window during which two policy versions are in force.

Pod Security Admission

The built-in Pod Security Admission controller enforces the Pod Security Standards, which define three levels: privileged (unrestricted), baseline (minimally restrictive), and restricted (heavily restrictive). It is configured with labels on the namespace. For each of three modes, enforce, audit, and warn, the label pod-security.kubernetes.io/<mode> sets a level, and an optional -version label pins the policy to a Kubernetes version. Enforce rejects violating Pods, audit records an annotation in the audit log, and warn shows a message to the client. The documentation notes that audit and warn also evaluate workload resources such as Deployments through their Pod templates, while enforce applies to Pods. That asymmetry has an operational consequence: a Deployment that violates an enforced level can be accepted and then fail to create Pods, so the error appears on the ReplicaSet’s events and not at the time of kubectl apply. The warn mode catches this earlier, which is a reason to enable it alongside enforce.

Exemptions can be configured for usernames, runtime classes, and namespaces. Every exemption is a hole in the isolation design, so keep the list short and review it.

ValidatingAdmissionPolicy

For custom rules, ValidatingAdmissionPolicy uses the Common Expression Language to validate requests inside the API server without an external service, and the documentation shows it as stable in Kubernetes 1.30. A policy has up to three parts: the ValidatingAdmissionPolicy with match rules and CEL expressions, an optional parameter resource, and a ValidatingAdmissionPolicyBinding that attaches the policy to namespaces or objects and chooses how a violation is handled. The binding’s validationActions can be Deny, Warn, and Audit. The documentation says Deny and Warn may not be used together. That allows a staged rollout of a rule: bind it with [Warn, Audit] first to see what would be rejected, then switch to [Deny].

The following illustrative policy requires every container in a Deployment to set a memory limit. It is an example written for this post, and its CEL expression was not evaluated against an API server.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-memory-limit.example.com
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments"]
  validations:
    - expression: >-
        object.spec.template.spec.containers.all(c,
          has(c.resources) && has(c.resources.limits) && 'memory' in c.resources.limits)
      message: "Every container must set a memory limit."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-memory-limit-tenants.example.com
spec:
  policyName: require-memory-limit.example.com
  validationActions: ["Warn", "Audit"]      # change to ["Deny"] after reviewing warnings
  matchResources:
    namespaceSelector:
      matchLabels:
        tenant: "true"

The binding’s namespace selector applies the rule to namespaces labeled tenant: "true" only, which keeps it away from system namespaces. That choice is deliberate. A policy that applies to every namespace, with a failure policy of Fail, can block the cluster’s own components when they are redeployed. The failurePolicy field says what happens when the policy itself cannot be evaluated, for example because of an error in the expression. Fail rejects such requests and Ignore lets them through. The documentation adds that failures defined by the failure policy are enforced according to the binding’s actions only if the policy is Fail (or unspecified), and are otherwise ignored, so an Ignore policy that cannot be evaluated enforces nothing.

Webhooks and their failure modes

Webhooks remain the tool when a rule needs data from outside the request, such as an inventory system, or when it mutates objects. They add a network dependency to every matching API request. If the webhook service is down and its failure policy rejects requests, the matching create and update operations fail cluster-wide. If the policy ignores failures, the rule is silently not enforced during the outage. Either behavior is an operational decision: set timeouts, scope the webhook’s rules and namespace selectors narrowly, exclude the namespace where the webhook itself runs, and monitor the webhook’s availability as part of the control plane. The glossary entry on blast radius makes the general point that a webhook’s rules, selectors, and failure policy determine how far its failure reaches.

A tenant baseline as one reviewed bundle

The settings above work best as a set that every tenant namespace receives. The next manifest is an illustrative baseline for team-a. It combines Pod Security labels, a quota, a LimitRange, a default-deny network policy with a DNS allow, and a namespaced role binding. The group name, quota values, and limits are assumptions for the example.

apiVersion: v1
kind: Namespace
metadata:
  name: team-a
  labels:
    tenant: "true"
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/warn: restricted
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute
  namespace: team-a
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "100"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: team-a
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 250m
        memory: 256Mi
      default:
        cpu: 500m
        memory: 512Mi
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: team-a
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: team-a
spec:
  podSelector: {}
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: team-a-edit
  namespace: team-a
subjects:
  - kind: Group
    name: team-a-developers          # placeholder group from your identity provider
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: edit
  apiGroup: rbac.authorization.k8s.io

The bundle has not been applied. Several details are worth checking before use: the LimitRange default limit of 500m CPU is above the default request of 250m, so the example does not trigger the request-above-default-limit rejection described earlier; the edit role is a built-in ClusterRole bound only within the namespace, which limits it to that namespace; and the DNS allow selects the DNS Pods by the k8s-app: kube-dns label, the label the DNS debugging guide uses. Distributions may label their DNS Pods differently, so check yours. The role and bundle should live in version control and be applied by a controller or pipeline, so that a new namespace cannot exist without them.

Version compatibility: what must move together

A shared cluster’s upgrade affects every tenant at once, so the upgrade rules deserve more attention than the cluster’s day-to-day settings.

Support windows

The Kubernetes version skew policy lists the three most recent minor releases as maintained. The patch release page describes a series as supported for about 12 months of standard patch releases followed by a two-month maintenance period, after which the series reaches end of life. A cluster that does not upgrade for more than that period is running a version that no longer receives fixes, including security fixes, from the project. Managed services publish their own support timelines, which may differ from the project’s, and some offer extended support for an additional cost. Treat the provider’s published lifecycle as the one that governs a managed cluster.

Skew between components

The skew policy states the combinations that are supported. The next table lists them using the documentation’s rules at the version it describes.

Component	Allowed relative to kube-apiserver
kube-apiserver instances in a highly available control plane	Newest and oldest within 1 minor version of each other
kubelet	Not newer than the API server; up to 3 minor versions older (2 for versions before 1.25)
kube-proxy	Not newer than the API server; up to 3 minor versions older
kube-controller-manager, kube-scheduler, cloud-controller-manager	Not newer than the API server; expected to match it or be 1 minor version older
kubectl	Within 1 minor version of the API server, older or newer

The policy’s upgrade order follows: the API server first, then the controller manager, scheduler, and cloud controller manager, then kubelets and kube-proxy, which may lag within the skew limit. The script below applies the kubelet rule. It is a model of the documented rule for minor versions only.

def check(apiserver, kubelet):
    if kubelet > apiserver: return "INVALID: kubelet newer than kube-apiserver"
    if apiserver - kubelet > 3: return "INVALID: kubelet more than 3 minors behind"
    return "ok"

With the API server at minor version 37 and an illustrative range of kubelet versions, the output was the following (the script’s output prints the full version string).

apiserver 1.37 kubelet 1.37: ok
apiserver 1.37 kubelet 1.36: ok
apiserver 1.37 kubelet 1.35: ok
apiserver 1.37 kubelet 1.34: ok
apiserver 1.37 kubelet 1.33: INVALID: kubelet more than 3 minors behind
apiserver 1.37 kubelet 1.38: INVALID: kubelet newer than kube-apiserver

The same script, applied to a second scenario, shows the practical limit of lagging nodes. With control-plane steps from 1.34 to 1.37, nodes left on 1.33 were valid through control-plane version 1.36 and became invalid at 1.37. Nodes left on 1.34 stayed valid through every step. The result is the documented skew rule applied to assumed versions. It shows that a team can defer node upgrades for a few control-plane steps, but not indefinitely, and that sequential minor upgrades of the control plane are the part that cannot be skipped. The kubeadm upgrade documentation states that minor versions cannot be skipped in a kubeadm upgrade. Whether a given provider lets you move nodes more than one minor version at a time must be checked in that provider’s documentation, because the skew policy permits the combination while node-upgrade tooling may impose its own rules.

What the skew table does not cover

The skew policy covers Kubernetes components. A cluster also runs add-ons whose compatibility is stated by their own projects: the network plugin, the storage driver and snapshot controller, the ingress controller, the metrics and logging agents, the admission webhooks, and any operators with custom resources. We infer from how these components call the Kubernetes API that each has a supported range of Kubernetes versions, so an upgrade plan needs a compatibility check for each one against the target version. The version skew table gives no assurance for them. Also, the documentation on kube-proxy notes that the default proxy mode is expected to change in a future version and recommends setting the mode explicitly, which is a concrete example of a default that can change during an upgrade.

API deprecation

Workload manifests are a compatibility surface too. The deprecation policy in the documentation sets minimum lifetimes: a generally available API version must remain served for at least 12 months or 3 minor releases after deprecation is announced, whichever is longer, and a beta version for at least 9 months or 3 releases after deprecation. Alpha versions have no guaranteed lifetime and can be removed in any release. The policy also says objects stored in etcd must stay readable, while serving an API version through REST can be disabled on the schedule. The consequence is that a manifest in version control that uses a removed API version keeps working until the day the cluster is upgraded past removal, and then the apply fails. The Kubernetes deprecation guide lists removals by version. Before each upgrade, compare the manifests and the Helm charts or controllers that generate them with the removal list for the target version. Whether your cluster also exposes a metric that counts requests to deprecated APIs is something to look up for your version, as the guide page read for this post did not describe one.

Staged upgrades

Staging an upgrade means making each step small, observable, and stoppable, and ordering the steps so that failure appears first where it costs least.

The order of operations

For a cluster created with kubeadm, the documentation gives this sequence: upgrade the first control plane node, then the other control plane nodes one at a time, then the worker nodes. The kubeadm page also says to read the release notes and to back up important components, especially etcd, before starting. It states that draining is mandatory when upgrading the kubelet to a new minor version: you must drain the node first. For managed services the steps are provider operations, but the same order applies conceptually, and the provider’s documentation states what it performs for you and what it leaves to you, such as node pool upgrades.

A staged plan commonly has these stages.

1. Rehearse the upgrade on a non-production cluster that has the same add-ons and a copy of representative manifests. This finds removed APIs and add-on incompatibilities without affecting tenants.
2. Upgrade the control plane. The control plane is replicated in a highly available cluster, and kubeadm upgrades control plane nodes one at a time, so the API remains available through most of the process. A single-control-plane cluster has a window when the API is unavailable. Running workloads typically continue, but nothing can be created or rescheduled during that window, which is an inference from how the control plane and kubelets divide work, so confirm it against your setup.
3. Upgrade a small canary set of nodes: one node, or a node pool that hosts low-risk workloads. Observe the node, the Pods rescheduled onto it, and tenant error rates before continuing.
4. Upgrade the remaining nodes in batches sized by the disruption budget, and pause between batches.
5. Verify the cluster’s behavior against the isolation controls, because upgrades can change defaults: run the cross-namespace network test, create an unlabeled Pod in a tenant namespace, and confirm the admission policy still responds.

The canary deployment glossary entry makes two points that apply to node canaries. A small share is an exposure limit, not evidence of correctness, so define the metrics, observation time, and stop conditions before starting. And a canary that cannot be stopped is not a canary, so write down the stop condition and who can invoke it.

Drain, disruption budgets, and the deadlock

Draining a node cordons it, so no new Pods are scheduled there, and evicts its Pods through the Eviction API, which honors PodDisruptionBudgets. A PodDisruptionBudget limits how many Pods of an application may be unavailable because of voluntary disruptions, expressed as minAvailable or maxUnavailable. The disruptions documentation clarifies two limits. A budget does not prevent involuntary disruptions such as hardware failure, though those count against the budget. And it does not constrain a Deployment’s or StatefulSet’s own rolling update, which is governed by the workload’s update settings. Percentages for minAvailable round up, per the documentation.

The number of Pods the Eviction API will allow to be evicted at a moment is the number of currently healthy Pods minus the number that must remain. The script below computes that for a few settings.

import math
def allowed_disruptions(expected, healthy, min_available=None, max_unavailable=None):
    desired = min_available if min_available is not None else expected - max_unavailable
    return max(0, healthy - desired)

The output lists replicas, healthy Pods, the budget, and the number of evictions allowed.

replicas healthy minAvail maxUnavail -> allowed evictions
       3       3 3        None       -> 0
       3       3 2        None       -> 1
       3       2 2        None       -> 0
       3       3 None     1          -> 1
       3       2 None     1          -> 0
       1       1 None     0          -> 0
       5       5 4        None       -> 1

Read the table as arithmetic over the documented definition of a budget, not as a simulation of a drain. Three rows matter. A budget of three available for three replicas permits no eviction, so a drain that must evict one of those Pods will wait, and the documentation says such a drain can fail. A budget of two of three permits one eviction at a time, but only while all three are healthy: when one Pod is already unhealthy, zero evictions are allowed, so an upgrade that proceeds quickly can stall until the replacement is ready. And a single-replica workload with maxUnavailable: 0 can never be drained without someone first making a decision about it, which the documentation lists as the approach for single-instance stateful applications.

The documentation recommends setting the unhealthyPodEvictionPolicy field to AlwaysAllow so that Pods that are not healthy can still be evicted during a drain. Without it, a crash-looping Pod can hold a node hostage. This is a trade: you gain the ability to drain nodes that host broken Pods and accept that evicting an unhealthy Pod does not count against availability.

Stateful workloads during node maintenance

The previous post’s StatefulSet behavior shapes node maintenance. For a three-member database with a budget of maxUnavailable: 1, a drain evicts one member, the StatefulSet recreates it, and the new Pod must attach the same volume and become ready before another member may be evicted. If the volume is zone-bound and the node pool lacks capacity in that zone, the replacement stays pending, and the budget keeps the drain from touching a second member. That is the budget doing its job, and the operational signal is a drain that does not complete. The cause is found in the Pod and claim events. Do not respond by force-deleting the Pod: as the previous post explained, force deletion removes the identity guarantee, and on an unreachable node it can leave two processes using one identity. Node maintenance for stateful workloads also depends on the data: the upgrade window should follow a restore-tested backup.

Time and capacity

Upgrades also consume capacity. Replacing nodes means running old and new nodes together, so the cluster needs headroom, and tenant quotas should leave room for rescheduling. The capacity planning entry describes capacity planning as a continuing cycle. Here its relevance is concrete: without spare capacity, a drain moves Pods onto fewer nodes and can cause the very contention the isolation controls were designed to limit.

Rollback: what can be undone

Rollback is not one operation. The glossary entry on rollback describes returning to a retained version and warns that the older version must still work with current data and interfaces. Each layer of a cluster has a different answer.

Layer	Can it be rolled back?	How, and what stays behind
Deployment	Yes	kubectl rollout undo restores a previous ReplicaSet revision; only Pod template changes create revisions, and the default history limit is 10 revisions
Admission policy	Yes	Switch the binding from Deny to Warn, or remove it; objects already admitted stay as created
Quota and LimitRange	Yes	Change or delete; existing Pods are not changed
Node pool	Replaceable	Add nodes at the old version if the skew rules allow; drained Pods have already moved
Control plane version	Not supported by kubeadm	The documentation states there is no supported downgrade path; recovery means retrying the idempotent upgrade or restoring etcd from backup
Data written by new code	No	Needs a data-level recovery plan: snapshots, backups, or a forward fix

The table follows the Deployment and kubeadm documentation. The kubeadm upgrade page says that kubeadm upgrade is idempotent and can be re-run after a failure, that etcd backups made during the upgrade are in a directory under /etc/kubernetes/tmp, and that downgrading Kubernetes is unsupported. An etcd restore recovers cluster state only and does not restore volume contents, as the previous post covered, and, because it restores a point-in-time snapshot, objects created after the snapshot are not in the restored state. Treat it as disaster recovery, not as an undo button.

Because control plane rollback is not supported, the real rollback design for a control plane upgrade is preparation: a tested etcd backup, a rehearsal on a non-production cluster, and a decision about what the team does if the new version misbehaves. One option, which we describe as a design possibility rather than a documented procedure, is to build a second cluster at the new version and move workloads gradually, keeping the first as the fallback. That option is only as good as the ability to move stateful data and traffic, so it must be tested, and it raises the cost of running two clusters during the move. The glossary entry on migration fallback describes the preparation such a return path needs.

For rollouts, rollback has a limit that is easy to miss. A Deployment rollback reverts the Pod template, and the older version must be compatible with whatever the newer version already wrote. A schema migration that has run, a message in a new format on a queue, or a persisted state change is not undone by restoring the old image. Making data changes backward compatible, by adding the new form first and removing the old form in a later release, is the usual way to keep that rollback possible. A rollback plan that skips the data question is incomplete.

Noisy neighbors: detection and response

Preventive controls reduce the chance of interference. They do not remove it, so a shared cluster needs a way to notice and respond. The noisy neighbor glossary entry gives the key point about detection: signals should be per tenant, because averages across tenants hide one tenant’s bad period.

Detection

The symptom from a victim’s side is inconsistency, such as the same request being fast at some times and slow at others. From the platform’s side, useful signals include per-namespace CPU throttling and memory working set against requests and limits, Pod restarts with out-of-memory termination reasons, node pressure conditions, scheduling delays, API server request latency and rejection counts per flow, and storage and network latency by volume or node. The specific metric names depend on the monitoring stack and the exporter versions, and this post does not claim any. The design requirement is that every metric is labeled with a tenant identifier, which the namespace usually provides, so you can ask which tenant changed when another tenant slowed down.

Response

A response runbook has to separate three actions that differ in speed and cost.

1. Identify and contain. Find the tenant and workload using the contended resource. Contain by lowering its parallelism, suspending the job, or reducing its quota, with the tenant owner informed. Containment has a cost to the contained tenant, and tenants should know the limits that apply, because a surprising throttle looks like an outage.
2. Protect the important work. Priority classes and the QoS rules decide who is evicted under node pressure and who preempts whom. The runbook should state which workloads have which priority and who may assign it, backed by quota on priority classes.
3. Fix the cause. Add requests and limits, move workloads onto separate node pools with taints and tolerations (the multi-tenancy page lists node isolation, noting that it reduces cost efficiency), add capacity, or change the schedule of the job. Each fix has a cost, and the choice should follow the objectives of the affected services.

Node isolation illustrates the trade. Dedicated nodes for a tenant remove CPU and memory interference on the node, but the nodes sit partly idle when the tenant is quiet, and a dedicated pool still shares the control plane, the network fabric, and possibly storage. Whether that costs more than shared nodes depends on sizing and pricing, and the comparison needs the same workload and objectives. The shared control plane remains shared in every namespace-based design, and its isolation depends on API Priority and Fairness and on quotas for object counts.

An operating model: who decides and who acts

Technical controls need owners. The next table assigns the responsibilities that the sections above create, using role names that are assumptions for this example. Adapt them to your organization.

Decision or action	Owner	Evidence it needs
Tenant onboarding bundle (namespace, quota, policy, RBAC)	Platform team	Bundle in version control; automated apply
Quota size and headroom for rollouts	Platform team with tenant owner	Historical usage; rollout surge calculation
Admission rule introduction (Warn, then Deny)	Platform team	Warn and audit results; exception process
Exceptions to a rule	Platform owner approves; tenant owner requests	Written reason, scope, expiry
Upgrade calendar and version target	Platform team	Support window; add-on compatibility; rehearsal result
Tenant readiness for upgrade (API versions, PDBs, replicas)	Tenant owner	Manifest scan; budget review
Stop decision during a staged upgrade	Named on-call for the platform	Predefined stop conditions
Noisy neighbor containment	Platform on-call, tenant informed	Per-tenant metrics

Two details keep this from being a paper exercise. First, the tenant’s duties need tooling: a manifest check against the removed-API list and a report of budgets that would block drain, delivered before the upgrade date, not discovered during it. Second, exceptions need expiry. A policy with permanent exceptions drifts into the state where the policy no longer describes the cluster. The next post compares Kubernetes with managed and simpler runtimes, and these operating costs are the main input to that comparison: choosing Kubernetes, managed services, or simpler runtime options.

References

• Kubernetes Documentation, Using RBAC Authorization
• Kubernetes Documentation, Namespaces
• Kubernetes Documentation, Multi-tenancy
• Kubernetes Documentation, Resource Quotas
• Kubernetes Documentation, Limit Ranges
• Kubernetes Documentation, Resource Management for Pods and Containers
• Kubernetes Documentation, Pod Quality of Service Classes
• Kubernetes Documentation, Deployments
• Kubernetes Documentation, API Priority and Fairness
• Kubernetes Documentation, Pod Priority and Preemption
• Kubernetes Documentation, Network Policies
• Kubernetes Documentation, Admission Controllers Reference
• Kubernetes Documentation, Pod Security Admission
• Kubernetes Documentation, Validating Admission Policy
• Kubernetes Documentation, Version Skew Policy
• Kubernetes Documentation, Patch Releases
• Kubernetes Documentation, Deprecation Policy
• Kubernetes Documentation, Deprecated API Migration Guide
• Kubernetes Documentation, Upgrading kubeadm Clusters
• Kubernetes Documentation, Disruptions
• Kubernetes Documentation, Specifying a Disruption Budget for your Application
• Kubernetes Documentation, Safely Drain a Node
• Kubernetes Documentation, Operating etcd Clusters for Kubernetes