There Is No Deny Rule: Isolation, Policy, and Upgrades on a Shared Cluster

A platform team onboards a new team the way the runbook says. A namespace of their own. A RoleBinding that grants the built-in edit role. A ResourceQuota so they cannot take the whole cluster. The restricted Pod Security label. A NetworkPolicy that denies ingress by default. Six months later an audit finds three things: that team’s CI identity can read Secrets belonging to another team, the egress everyone believed was blocked has been flowing the entire time, and the cluster cannot be upgraded because some nodes are three minor versions behind. This scenario is hypothetical, but each of the three outcomes follows directly from how the mechanisms are documented to behave.

The word “isolation” suggests a wall that somebody built. On a shared Kubernetes cluster, the two mechanisms that decide who can reach what have no way to say no. RBAC permissions are “purely additive (there are no ‘deny’ rules).” NetworkPolicy works the same way once it applies to a pod: its model is “deny by default, with only the ability to add allow rules,” and “the ability to explicitly deny policies” appears on the documentation’s own list of things the API cannot do. (Before any policy selects a pod, nothing is denied at all — that is the next distinction, and it matters.)

So a boundary here is not a line. It is the union of everything that was allowed, computed across objects that different people wrote at different times. And the union moves without anyone touching it: a wildcard permission absorbs resources added in a later release, the Pod Security policy itself is versioned, beta APIs stop being served on a published schedule, and the components can only drift so far apart before the upgrade path closes.

Two things do reject. Admission-time gates say no at creation: a quota violation comes back as “HTTP status code 403 Forbidden,” and Pod Security in enforce mode causes “the pod to be rejected.” Those are conditions on making an object, not denials of reach. Keeping the two kinds of mechanism separate in your head is most of the skill. Quotations here were read in September 2026 against the v1.37 documentation; skew windows, support periods and policy levels change with releases.

A tenant is not a Kubernetes object

Start with what the platform does not give you. “While Kubernetes does not have first-class concepts of end users or tenants, it provides several features to help manage different tenancy requirements.” And there is “no single definition for a ‘tenant’” — the word means different things depending on which situation you are in.

The documentation splits multi-tenancy into two cases, and the split decides what work you actually do.

• Multiple teams. Tenants hold API access, directly or through release automation. “There is often some level of trust between members of different teams, but Kubernetes policies such as RBAC, quotas, and network policies are essential to safely and fairly share clusters.” Your threat model is dominated by mistakes made by trusted people who can call the API, so the control plane is the main lever.
• Multiple customers. “The customers do not have access to the cluster; Kubernetes is invisible from their perspective and is only used by the vendor to manage the workloads.” Nobody outside operates the API, so the work moves to the data plane and to cost.

Resist the urge to ask whether your cluster is “hard” or “soft” multi-tenant. The documentation says plainly that the “terms hard and soft can often be confusing, as there is no single definition that will apply to all users. Rather, ‘hardness’ or ‘softness’ is better understood as a broad spectrum.” The useful question is narrower and answerable: what does this specific mechanism prevent, and what does it leave open?

Isolation applies in two places, and they fail independently. Control plane isolation “ensures that different tenants cannot access or affect each others’ Kubernetes API resources.” Data plane isolation “ensures that pods and workloads for different tenants are sufficiently isolated.” The data plane is the harder half: “since data planes typically have much larger attack surfaces, ‘hard’ multi-tenancy often requires extra attention to isolating the data-plane, though control plane isolation also remains critical.”

The namespace is where most designs start, and it gives you exactly two things: object names can overlap between namespaces, and “many Kubernetes security policies are scoped to namespaces.” Note the word scoped. A namespace is where policies can attach; it is not itself a policy. The documentation states the dependency directly: “the namespace isolation model requires configuration of several other Kubernetes resources, networking plugins, and adherence to security best practices to properly isolate tenant workloads.” An empty namespace isolates names and nothing else. (This is the Kubernetes namespace, an API grouping — not the Linux kernel namespace that isolates a single container‘s view of the system. Same word, different layer.)

One recommendation is worth adopting early because it is cheap now and expensive later: “a common practice is to isolate every workload in its own namespace, even if multiple workloads are operated by the same tenant,” and name them so they are “unique across your entire fleet (that is, even if they are in separate clusters).” Fleet-unique names are what let you move a tenant from a shared cluster to a dedicated one without rewriting their manifests.

Permissions add up, and nothing subtracts them

Among all the policies you can apply, one comes first. “The most important type of isolation for the control plane is authorization. If teams or their workloads can access or modify each others’ API resources, they can change or disable all other types of policies thereby negating any protection those policies may offer.” Every quota, label and network policy in this article is an object in the API. Whoever can edit those objects has already won.

RBAC has four object kinds — Role and RoleBinding inside a namespace, ClusterRole and ClusterRoleBinding outside it — and one property that shapes everything: permissions are additive with no deny. You cannot narrow an over-broad grant by adding a restriction. You have to find the grant and remove it.

Which makes the built-in roles worth reading literally rather than by name. The single most consequential sentence in the RBAC documentation is about edit: it “allows read/write access to most objects in a namespace,” and then: “However, this role allows accessing Secrets and running Pods as any ServiceAccount in the namespace, so it can be used to gain the API access levels of any ServiceAccount in the namespace.”

Unpack that in steps, because it is the mechanism behind the first finding in the opening scenario.

1. A holder of edit can create a Pod and set its service account to any service account in that namespace.
2. That Pod receives the service account’s credentials.
3. So the holder can act with every permission any workload identity in that namespace holds — including permissions granted cluster-wide by a ClusterRoleBinding.
4. If a controller, operator or CI agent lives in that namespace with a ClusterRole that reads Secrets across the cluster, then edit in that one namespace is effectively that ClusterRole.

Nobody granted cross-tenant Secret access. It is the union of two grants that were each defensible alone. The same logic explains why view deliberately excludes Secrets: “reading the contents of Secrets enables access to ServiceAccount credentials in the namespace, which would allow API access as any ServiceAccount in the namespace (a form of privilege escalation).” Read access to secrets is not read access; it is impersonation.

Built-in role	What it says	The part people miss
view	Read-only on most objects	Excludes Secrets on purpose, and “does not allow viewing roles or role bindings”
edit	Read/write on most objects	Grants the API access of any ServiceAccount in the namespace
admin	Namespace administration via RoleBinding	Includes “the ability to create roles and role bindings within the namespace,” but “does not allow write access to resource quota or to the namespace itself”
cluster-admin	“Super-user access to perform any action on any resource”	In a RoleBinding it gives “full control over every resource in the role binding’s namespace, including the namespace itself“

The admin carve-out is a deliberate design worth copying: a tenant can manage their own permissions but cannot raise their own quota. Delegation of authority and allocation of capacity are different decisions with different owners.

Now the part where the boundary moves on its own. Wildcards look like convenience and behave like a standing subscription: “if a new resource type is added, or a new subresource is added, or a new custom verb is checked, the wildcard entry automatically grants access, which may be undesirable.” A role written as all resources and all verbs in an API group grants, after your next upgrade, whatever that group gained in the meantime. Nobody edited the role. This is least privilege decaying by default.

One mechanism here does act like a denial, and it will surprise you by constraining you. “The RBAC API prevents users from escalating privileges by editing roles or role bindings,” and “you can only create/update a role if… you already have all the permissions contained in the role, at the same scope,” unless you hold explicit permission for the escalate verb. The reach of this rule is broader than the authorizer: “because this is enforced at the API level, it applies even when the RBAC authorizer is not in use.” So a perfectly legitimate delegation can fail with a confusing error simply because the person or pipeline applying it lacks the permissions it contains.

A practical trap for teams running GitOps: aggregated ClusterRoles have their rules filled in by a controller, and “the control plane overwrites any values that you manually specify in the rules field.” Leaving a rules field in the manifest, “even to an empty list, claims ownership of the field when the manifest is applied with server-side apply,” after which applies “either fail with a field manager conflict on .rules, or (if conflicts are forced, as GitOps controllers typically do) repeatedly clear the aggregated rules, which the control plane then fills in again.” The symptom is permissions that disappear and come back on a cycle. Omit the field.

The operational conclusion is about the shape of the review, not its frequency. Auditing permissions means computing a union across three layers: the bindings, the roles they point at, and the identities those roles let you become. A list of RoleBindings answers none of it. If you cannot expand the third layer, you do not know what you granted.

A network policy you cannot tell is working

The network starts fully open. “By default, all pods in a Kubernetes cluster are allowed to communicate with each other, and all network traffic is unencrypted.” Restricting that is the job of a network policy, which selects pods by label and allows connections at the IP and port level.

Before any of the semantics, the prerequisite that produces the second finding in the opening scenario. “Network policies are implemented by the network plugin,” and “creating a NetworkPolicy resource without a controller that implements it will have no effect.” The multi-tenancy guidance repeats it as a warning: without a CNI plugin that supports them, “NetworkPolicy resources will be ignored.”

Sit with what that means. The object is accepted by the API server. It appears in listings. It passes every review. Nothing in the API distinguishes a policy that is enforced from one that is decoration, and the API has no way to log a blocked connection either — “the ability to log network security events (for example connections that are blocked or accepted)” is on the list of things NetworkPolicy cannot do. So the first check on any cluster is not a manifest review but an actual connection attempt that should fail. Absent that test, you have a belief, not a control. Contrast the API server, where an audit log exists by design.

The semantics are simple once you accept that there is no deny. Isolation is per-direction and conditional: “by default, a pod is non-isolated for egress; all outbound connections are allowed,” and it becomes isolated “if there is any NetworkPolicy that both selects the pod and has ‘Egress’ in its policyTypes.” Even the word is qualified — “‘isolation’ here is not absolute, rather it means ‘some restrictions apply’.”

From there, four consequences that determine how you debug.

• Order does not matter, and nothing conflicts. “Network policies do not conflict; they are additive.” Where several apply, “the connections allowed in that direction from that pod is the union of what the applicable policies allow. Thus, order of evaluation does not affect the policy result.” There is no precedence to reason about — and no way to carve an exception out of an allowance.
• Both ends must agree. “For a connection from a source pod to a destination pod to be allowed, both the egress policy on the source pod and the ingress policy on the destination pod need to allow the connection.” A blocked connection between two teams is a two-namespace investigation.
• Omitting policyTypes is a decision. “If no policyTypes are specified… by default Ingress will always be set and Egress will be set if the NetworkPolicy has any egress rules.” A policy written to restrict inbound traffic leaves outbound completely open, which is exactly how the opening scenario’s egress kept flowing.
• Some edges are never closed. “A pod cannot block access to itself,” and “traffic to and from the node where a Pod is running is always allowed, regardless of the IP address of the Pod or the node.” A deny-all is also protocol-limited: “it is only guaranteed to deny TCP, UDP and SCTP connections.”

The recommended starting posture is a default-deny, and it comes with a footgun the documentation flags in bold terms: “a default deny-all egress policy also blocks DNS traffic. If your workloads need DNS resolution, you must add a separate NetworkPolicy that allows egress to your cluster’s DNS.” The multi-tenancy page gives the same recipe from the other side — start by denying pod-to-pod communication “with another rule that allows all pods to query the DNS server for name resolution” — and adds one more: do not use an empty label selector for namespaceSelector, which silently means every namespace.

Two timing behaviours are worth designing around rather than discovering. At startup, a pod “may be started unprotected” if the plugin has not finished handling a policy yet, and once handling begins the isolation rules can land before the allow rules, so “in the worst case, a newly created pod may have no network connectivity at all when it is first started.” There is “no way to tell from the Kubernetes API when exactly that happens,” from which the documentation draws an explicit requirement: “pods must be resilient against being started up with different network connectivity than expected.” An init container that waits for a destination to be reachable is the documented remedy — and a process that exits on its first failed connection is the anti-pattern.

At the other end, revocation is not immediate and not specified. When policies change for an established connection, “it is implementation defined as to whether the change will take effect for that existing connection,” and the advice is “not to modify policies/pods/namespaces in ways that might affect existing connections.” A network policy is admission control for new connections, not a kill switch. If you need to cut off a compromised workload now, delete the Pod.

Finally, what the API deliberately does not cover, as of the v1.37 documentation: no explicit deny, no targeting Services by name, no node-identity policies, no policies that apply across all namespaces by default, and no event logging. Those gaps are the reason service meshes and third-party policy engines exist, and the reason a policy layer is something you assemble rather than enable. Expressing the assembled rules as reviewed, versioned files — policy as code — is what keeps the union inspectable.

Quotas are accounting, not enforcement

Fairness is a separate problem from access, and ResourceQuota is the instrument: it “provides constraints that limit aggregate resource consumption per namespace,” and can also cap object counts by API kind. Violations are rejected at admission with “HTTP status code 403 Forbidden.”

What it is not is a guarantee, and four documented properties explain why.

• It is not retroactive. “Neither contention nor changes to quota will affect already created resources.” Lowering a quota reclaims nothing; it only constrains the next creation. A tenant that is already over a new limit stays over it until something restarts.
• It is unaware of your cluster. “ResourceQuotas are independent of the cluster capacity. They are expressed in absolute units. So, if you add nodes to your cluster, this does not automatically give each namespace the ability to consume more resources.” New capacity does not distribute itself.
• It can be promised twice. “In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces, there may be contention for resources. This is handled on a first-come-first-served basis.” Oversubscription is allowed, and when it bites, the tiebreaker is arrival order rather than importance.
• It does not separate neighbours. “Resource quota divides up aggregate cluster resources, but it creates no restrictions around nodes: pods from several namespaces may run on the same node.” A quota bounds the bill, not the noisy neighbour.

Turning a quota on also changes the rules for everyone’s manifests: “if you enforce a resource quota in a namespace for either cpu or memory, you and other clients, must specify either requests or limits for that resource, for every new Pod you submit. If you don’t, the control plane may reject admission for that Pod.” So enabling a quota can break workloads that were valid yesterday. The documented cushion is a LimitRange, “to force defaults on pods that make no compute resource requirements” — which is how you get the accounting without an onboarding tax on every team.

And a failure mode that is easy to misdiagnose because the error is not where you are looking: “if you create a Deployment that tries to use more resources than are available, the creation of the Deployment (or other workload management object) succeeds, but the Deployment may not be able to get all of the Pods it manages to exist.” The apply returns success. The rollout stalls. The rejection is recorded against the ReplicaSet, one level below where the author is watching. That shape — accepted at the top, rejected at the bottom — repeats in the next section, and it is a direct consequence of declared state being reconciled by a control loop rather than executed by your command.

One more instruction that ties back to permissions: “the cluster administrator should also restrict access to delete or update that ResourceQuota.” A limit that its subject can edit is a suggestion — which is precisely why the built-in admin role stops short of quota writes.

Where genuine workload isolation is needed, quotas are the wrong tool and the documentation says so, pointing at node isolation instead. Quotas prevent one tenant from consuming “greater than their allocated share,” which limits resource exhaustion — but “quotas cannot protect against all kinds of resource sharing, such as network traffic.”

The control plane has its own scarcity, handled by a different mechanism. With API Priority and Fairness, “calls from pods with higher priority are fulfilled before those with a lower priority. When contention is high, lower priority calls can be queued until the server is less busy or you can reject the requests.” A tenant’s controller in a tight reconcile loop is a noisy neighbour in the API server, and request throttling there is the protection — a different meaning of the word from the kernel delaying a cgroup.

What a workload may ask of the node

Requests and quotas govern how much. Pod Security governs what kind — specifically, which fields a Pod may set to reach past its own boundary. The Pod Security Standards define three levels that are “cumulative and range from highly-permissive to highly-restrictive.”

Level	Documentation’s description	Use
Privileged	“Unrestricted policy… This policy allows for known privilege escalations“	System and infrastructure workloads run by trusted operators
Baseline	“Minimally restrictive policy which prevents known privilege escalations. Allows the default (minimally specified) Pod configuration”	Ordinary application workloads
Restricted	“Heavily restricted policy, following current Pod hardening best practices”	Anything you would not personally vouch for

Baseline is the interesting line because of what it names: sharing the host namespaces “must be disallowed,” which covers spec.hostNetwork, spec.hostPID and spec.hostIPC; privileged containers “disable most security mechanisms and must be disallowed”; and capabilities beyond a fixed list cannot be added. Those are the fields through which a container stops being separate from its node. Note also that judgement is per-Pod, not per-container: “if any of the listed containers fails to meet the requirements, the entire pod will fail validation.”

Enforcement is by namespace label, in three modes — enforce rejects the pod, audit writes “an audit annotation to the event recorded in the audit log,” warn returns “a user-facing warning” — and “a namespace can configure any or all modes, or even set a different level for different modes.”

Here is the sentence to internalize, because it produces the same confusing failure as the quota example: “both the audit and warning modes are applied to the workload resources. However, enforce mode is not applied to workload resources, only to the resulting pod objects.“

So a namespace configured with enforce alone accepts a Deployment whose template violates the policy, and then silently produces no Pods. The developer sees a successful apply and an empty rollout. Setting warn and audit alongside enforce is what moves the message to the moment of the mistake — enforce protects the cluster, warn protects the person, and you want both.

Exemptions behave asymmetrically and the documentation warns about both directions. They “must be explicitly enumerated” across three dimensions — usernames, runtime class names, namespaces — and then: “exempting an end user will only exempt them from enforcement when creating pods directly, but not when creating a workload resource,” while exempting a controller’s service account “would implicitly exempt any user that can create the corresponding workload resource.” Exempting a person usually does not work; exempting a controller usually works far too well.

And this is the second place where your boundary moves on its own. There is an optional per-mode version label that “can be used to pin the policy to the version that shipped with a given Kubernetes minor version,” or you can set latest. Leave it floating and an upgrade can change the verdict on a Pod nobody edited. Pin it and you have chosen to review policy changes deliberately instead of receiving them. Either is defensible; not knowing which you chose is not.

Three metrics on the API server make the state observable rather than assumed: pod_security_evaluations_total, pod_security_exemptions_total, and pod_security_errors_total — with a fail-safe worth knowing about, since “non-fatal errors may result in the latest restricted profile being used for enforcement.”

All of which has a ceiling. Pod Security reduces what a workload may ask for. It does not add a kernel. Containers “utilize OS-level virtualization and hence offer a weaker isolation boundary than virtual machines that utilize hardware-based virtualization”; they “are processes running on a shared kernel; they mount file systems like /sys and /proc from the underlying host.” For code you genuinely do not trust, the documented answer is a different execution environment, not a stricter label: sandboxing runs “each pod in a separate execution environment such as a virtual machine or a userspace kernel,” and is “often recommended when you are running untrusted code, where workloads are assumed to be malicious.” Treat an unpatched container breakout as a live vulnerability class rather than a theoretical one.

Between namespace policy and sandboxing sits node isolation: dedicate a set of nodes per tenant so that “co-mingling of tenant pods is prohibited,” implemented with taints plus “a mutating webhook… to automatically add tolerations and node affinities to pods deployed into tenant namespaces.” It has a real commercial advantage — “easier to reason about from a billing standpoint than sandboxing containers since you can charge back per node rather than per pod” — and a real limit: “the kubelet and (unless using virtual control planes) the API service are still shared services. A skilled attacker could use the permissions assigned to the kubelet or other pods running on the node to move laterally within the cluster.”

Two more cluster-wide leaks to close deliberately. Service discovery ignores your network policy: “by default, the Kubernetes DNS service allows lookups across all namespaces in the cluster,” so a tenant can enumerate everyone’s service names even when no packet gets through; CoreDNS can be configured to restrict queries to a namespace. And storage crosses the boundary by design, because “a PersistentVolume is a cluster-wide resource and has a lifecycle independent of workloads and namespaces” even though a PersistentVolumeClaim is namespaced.

That last one deserves a flag, because two correct recommendations point in opposite directions. For tenant separation: “if a StorageClass is shared, you should set a reclaim policy of Delete to ensure that a PersistentVolume cannot be reused across different namespaces.” For data safety, Delete is the policy that destroys the underlying disk when a claim goes away. Both statements are right, and they cannot both be satisfied by one shared StorageClass. The resolution the documentation points at is a StorageClass per tenant — “you can configure a separate StorageClass for each tenant and use this to strengthen isolation” — which lets the reclaim policy be a per-tenant decision instead of a cluster-wide compromise.

How far a mistake travels

Every choice so far has been about preventing something. The remaining question is what happens when prevention fails, which is the question of blast radius. AWS’s cell-based architecture guidance states the goal precisely: “fault isolated boundaries restrict the effect of a failure within a workload to a limited number of components. Components outside of the boundary are unaffected by the failure.” It even gives the arithmetic — “if a workload uses 10 cells to service 100 requests, when a failure occurs in one cell, 90% of the overall requests would be unaffected by the failure” — and names the failure types that are otherwise hard to contain: “unsuccessful code deployments or requests that are corrupted or invoke a specific failure mode.”

On a shared cluster, the practical version of that question is: which objects are not namespaced? Those are the ones with shared fate. The documentation names them when explaining the limits of namespace isolation — it “doesn’t apply to Kubernetes resources that can’t be namespaced, such as Custom Resource Definitions, Storage Classes, and Webhooks.”

Read that list as a risk register. A CRD is a schema every tenant shares. A StorageClass is a default that quietly decides whether disks survive. A validating or mutating webhook sits in the path of API requests cluster-wide, which means one team’s misconfigured webhook can stop every other team from creating anything — and if it was installed to implement node isolation or inject policy, it is now both a control and a single point of failure. The documentation’s own phrase for this class of problem is “uncontrollable blast radius of policy misconfigurations.”

Which gives a decision that is about degree of separation rather than about tools.

Level	What it contains	What it costs
Namespace per tenant	API resources, most policies. “Negligible resource cost”	“Can be difficult to configure,” and leaves CRDs, StorageClasses and webhooks shared
Virtual control plane per tenant	“Noisy neighbors in the control plane, uncontrollable blast radius of policy misconfigurations, and conflicts between cluster scope objects such as webhooks and CRDs”	“Running and maintaining an individual virtual control plane per tenant,” and it “does not solve isolation problems in the data plane”
Cluster per tenant	Nearly everything, up to “dedicated hardware if VMs are not considered an adequate security boundary”	“The cost and complexity of managing multiple clusters”

Note what the middle row does and does not buy. A virtual control plane per tenant fixes the shared-object problems exactly, and fixes nothing on the node: “per-tenant control planes do not solve isolation problems in the data plane, such as node-level noisy neighbors or security threats. These must still be addressed separately.” Stacking mechanisms is not redundancy when they protect different planes.

There is also a blast radius that has nothing to do with tenants: the one you create yourself every time you change something. The bulkhead idea — from ship design, where “vertical partition walls subdivide the ship’s interior into self-contained, watertight compartments” — applies to rollouts as much as to failures, which is the logic behind a canary deployment and behind keeping a rollback that actually works. And it applies to the platform itself: upgrading every node at once is a change with a cluster-wide radius, which is the subject of the rest of this article.

One honest caution about measuring this. Blast radius is a property of the architecture, not of the last incident. An outage that happened to stay small because the failing component was idle at 3am tells you nothing about the boundary. The question to answer on a whiteboard, before it is answered for you, is which shared object would have to fail for every tenant to notice — and whether that list is short enough to defend. Partial failure is the normal condition; the design question is only how partial.

The cluster you secured is not the cluster you will run

Everything above describes a configuration. Kubernetes does not let a configuration sit still, and the constraints are published rather than discovered, so this is plannable work rather than a surprise.

Start with the clock. “The Kubernetes project maintains release branches for the most recent three minor releases,” and versions from 1.19 on “receive approximately 1 year of patch support.” The patch release page is more precise about what that year contains: support runs “for a period of roughly fourteen (14) months,” of which “the first twelve months… will be considered the standard period,” followed by two months of maintenance mode where releases address only “vulnerabilities that have an assigned CVE ID,” “dependency issues (including base image updates)” and “critical core component issues,” after which the series is “EOL (end of life).” Patch cadence is “typically monthly.” So a minor release gives you roughly twelve months of ordinary support and a two-month grace period that is explicitly not for bug fixes. That is the shape of the support lifecycle you are budgeting against, and it means several minor upgrades per year is the steady state, not a project.

Then the distances. Versions are “expressed as x.y.z… following Semantic Versioning terminology,” and each component has a documented version skew it must stay within.

Component	Allowed relative to kube-apiserver
kube-apiserver (HA peers)	Newest and oldest “must be within one minor version“
kubelet	“Must not be newer”; “may be up to three minor versions older“
kube-proxy	Same as kubelet, and within three minors of the kubelet beside it
controller-manager, scheduler, cloud-controller-manager	“Must not be newer”; “may be up to one minor version older (to allow live upgrades)”
kubectl	“Within one minor version (older or newer)”

Those distances imply an order, which the documentation spells out: kube-apiserver first, then controller-manager, scheduler and cloud-controller-manager (in any order or “even simultaneously”), then kubelet, then kube-proxy. And a hard constraint that removes a tempting shortcut: project policies “require kube-apiserver to not skip minor versions when upgrading, even in single-instance clusters.” Three versions behind means three upgrades, in sequence.

Four consequences matter more than the table itself.

• Deferring becomes a block. “Running a cluster with kubelet instances that are persistently three minor versions behind kube-apiserver means they must be upgraded before the control plane can be upgraded.” The node work you postponed is now a prerequisite for a security patch — the third finding in the opening scenario, and the reason node upgrades should not be the flexible part of the plan.
• Every node upgrade disrupts workloads. “Before performing a minor version kubelet upgrade, drain pods from that node. In-place minor version kubelet upgrades are not supported.” So platform maintenance is repeated, deliberate Pod termination, and whether that is invisible or an incident depends on whether your workloads handle graceful shutdown properly.
• The middle of the upgrade is the tightest moment. When apiserver instances in an HA cluster are at different minors, that “narrows the allowed kubelet versions” and the supported kubectl range too. A partially upgraded control plane constrains everything around it, including your operators’ tooling.
• Your policy layer is a prerequisite. The requirement is that registered admission webhooks “are able to handle the data the new kube-apiserver instance will send them,” including new resource versions and new fields, with matchPolicy: Equivalent available as a cushion. The webhook you installed to enforce isolation is now something the upgrade waits on.

The documented preparation is unglamorous and reduces risk for free: get every component to the newest patch of its current minor first, then move to the newest patch of the target minor.

APIs leave on a schedule

Version skew tells you how far apart components may be. The deprecation policy tells you what disappears while you wait, and it is unusually specific about time — which makes it possible to plan against instead of reacting to.

The core rule is that APIs are removed by version, not in place: “API elements may only be removed by incrementing the version of the API group,” and once added to a version they cannot “be removed from that version or have its behavior significantly changed.” Lifetime then depends on the track. GA versions “may be marked as deprecated, but must not be removed within a major version of Kubernetes.” Beta versions “are deprecated no more than 9 months or 3 minor releases after introduction (whichever is longer), and are no longer served 9 months or 3 minor releases after deprecation (whichever is longer).” Alpha versions “may be removed in any release without prior deprecation notice.”

The stated reason connects the two policies: this “ensures beta API support covers the maximum supported version skew of 2 releases, and that APIs don’t stagnate on unstable beta versions, accumulating production usage that will be disrupted when support for the beta API ends.” The clock is not an inconvenience; it exists because unversioned adoption is how a breaking change becomes an outage.

Three other clocks catch people out, and they are easy to miss because each belongs to a different team.

• Admin-facing flags expire twice as fast. A deprecated GA element of a user-facing CLI such as kubectl must keep working “12 months or 2 releases (whichever is longer)”; for an admin-facing component such as the kubelet it is “6 months or 1 release (whichever is longer).” The flags baked into your node images have half the runway of the commands your developers type. Deprecated behaviours, separately, get “no less than 1 year.”
• Feature gates default differently by stage. “Alpha: the feature gate is disabled by default and can be enabled by the user.” Beta means “the feature gate is enabled by default and can be disabled by the user.” So a beta feature is on in your cluster unless someone turned it off, and when it graduates the gate itself is deprecated on its own schedule — “beta feature to GA: 6 months or 2 releases,” “beta feature to EOL: 3 months or 1 release,” alpha to EOL immediately.
• Metrics go quiet rather than loud. A deprecated STABLE metric keeps working “3 releases or 9 months (whichever is longer),” and then: “when the metric’s deprecatedVersion is equal to current_kubernetes_version – 3, a deprecated metric will become a hidden metric.” An alert whose metric has disappeared does not fail — it simply never fires again, which is the worst possible failure mode for a control you are relying on.

Preparing for an upgrade is therefore not a grep through manifests. It is keeping four inventories current, because each of them expires on a different clock: the API versions in use — including the ones your controllers, operators and CRDs use on your behalf, not just the ones in your YAML; the feature gates you have set; the metrics your alerts and dashboards read; and how your admission webhooks handle versions they have not seen. That is deprecation management as a standing practice rather than an upgrade-week scramble.

What the platform team owes

Pull it together into what someone actually has to do, because “we use namespaces and RBAC” is not an answer to any of the above.

Onboarding a tenant is a bundle, not a namespace. A fleet-unique namespace name; a RoleBinding chosen with the edit problem in mind, which usually means not granting edit in a namespace that also hosts controllers with broad ClusterRoles; a ResourceQuota the tenant cannot edit; a LimitRange so the quota does not break their manifests; Pod Security labels with enforce, warn and audit set and the policy version pinned deliberately; a default-deny NetworkPolicy plus the DNS allowance it requires; and a StorageClass whose reclaim policy was a decision rather than a default.

Verify, because most of these fail silently. Each item in that bundle has a specific way of being present and inert, so each needs its own proof.

• A connection that should be refused, attempted — the only way to know the plugin enforces policy at all, since nothing in the API says so and no event is logged.
• A deliberately violating Pod, created directly, to prove Pod Security enforce is active in that namespace, plus a violating Deployment to prove the author gets a warning.
• A Pod with no requests, to confirm the quota-and-LimitRange pair behaves as intended rather than rejecting everything.
• A permission review that expands bindings into roles into reachable identities, rather than listing bindings.
• pod_security_exemptions_total and the audit log as standing signals, since exemptions and violations are the things that accumulate quietly.

Decide who decides. Capacity and quota allocation is a funding decision, not an engineering one, which is why quota writes are withheld from tenant administrators. Every exemption is an accepted risk and should carry an owner and an expiry date, because an exemption with neither becomes permanent. Upgrade windows are negotiated against service objectives, and the negotiation is easier when the support clock is on the table: twelve months of standard support, then two months that only cover CVEs and critical fixes.

Know what would make you change the design. A tenant who cannot tolerate shared-fate objects — CRDs, StorageClasses, webhooks — is telling you that namespace isolation is the wrong level for them, and the answer is a virtual control plane or a separate cluster, not another policy. Code you would describe as untrusted needs a different execution environment, not a stricter label. And a cluster where nobody can produce the four upgrade inventories on request is already behind, whichever version it is running.

Which brings the opening scenario back with its mechanisms visible. The CI identity could read another team’s Secrets because edit in a namespace confers the API access of every identity in it. The egress flowed because a policy without policyTypes only restricts ingress — and, had the plugin lacked NetworkPolicy support, it would have flowed with the field set too. The upgrade was blocked because kubelets three minor versions behind must be upgraded before the control plane can move. Nothing malfunctioned, and nobody was careless. Each boundary was simply the union of what had been allowed, and nobody had computed the union. That computation, repeated on a schedule, is what operating a shared cluster consists of.

References, read September 2026 against the Kubernetes v1.37 documentation: Kubernetes: Multi-tenancy; Using RBAC Authorization; Network Policies; Resource Quotas; Pod Security Standards; Pod Security Admission; Version Skew Policy; Patch Releases; Kubernetes Deprecation Policy; and AWS Well-Architected: Reducing the Scope of Impact with Cell-Based Architecture. Skew windows, support periods, policy levels and feature maturity change between releases.