Kubernetes Isolation, Policy, and Upgrades

A platform team onboards another team with a namespace, an edit RoleBinding, a ResourceQuota, Restricted Pod Security and an ingress-deny NetworkPolicy. Later it discovers cross-team Secret access through a powerful service account, outbound traffic that no egress policy restricted, and nodes too old for the next control-plane minor upgrade. This hypothetical case depends on those additional conditions; the onboarding objects alone do not establish the outcome.

RBAC and standard NetworkPolicy use additive permissions: there is no explicit deny rule that overrides an applicable allowance. That does not mean they cannot refuse access. Requests without authorization are denied, and an isolated network direction permits only the connections its applicable policies allow, subject to documented exceptions.

Review the effective allowances across objects and identities, then the additional controls that constrain them. A wildcard can expand permission when new resources appear, while policy versions and supported component combinations can change during upgrades.

Admission adds another decision after authorization. ResourceQuota enforces covered limits, and Pod Security enforce rejects violating Pod requests within its scope, including applicable updates. These checks govern object changes; they do not replace API authorization or packet filtering. Check the policy and version implemented by the cluster.

A tenant is not a Kubernetes object

Kubernetes has no first-class tenant or end-user object. Define the tenant boundary from the people and workloads that share the cluster; tenancy requirements vary with that relationship.

The documentation splits multi-tenancy into two cases, and the split decides what work you actually do.

• Multiple teams. Tenants use the API directly or through release automation. Some trust may exist between teams, but RBAC, quotas, and network policies still protect fair and safe sharing. Control-plane access is a main lever for mistakes or misuse by API users.
• Multiple customers. In this model the vendor operates the cluster and customers do not use its API. The vendor must address workload separation in the data plane and the costs of that separation.

Treat hard and soft multi-tenancy as a spectrum rather than universally defined categories. Assess what each selected mechanism prevents and what remains shared.

Control-plane isolation protects tenants’ API resources from access or interference by others. Data-plane isolation separates their running workloads. The larger data-plane attack surfaces demand particular attention, while control-plane controls remain essential.

A namespace scopes object names and many security policies. Policies, networking plugins, and secure configuration must supply the workload boundary; an empty namespace primarily separates names. This API grouping differs from a Linux kernel namespace, which changes a container process’s resource view.

Fleet-unique namespace names can simplify identifying and relocating workloads. They do not guarantee that manifests transfer unchanged: storage, identity, networking and cluster-level dependencies may still require migration work.

Additive permissions and effective access

Control-plane authorization protects the controls themselves. Quotas, labels, and network policies are API objects: an identity able to modify another tenant’s controls may weaken their protection. Inspect effective rights before relying on those objects.

Roles and RoleBindings are namespaced; ClusterRoles and ClusterRoleBindings are cluster-scoped. A RoleBinding can reference a ClusterRole but grants its applicable permissions within the binding’s namespace. Because RBAC grants add together, narrowing access requires removing the relevant grants rather than adding a deny rule.

Read built-in roles by effective permission rather than by name. edit grants read/write access to most namespace objects, including Secrets and the ability to run Pods as any ServiceAccount in that namespace. Those Pod rights can expose the chosen service account’s API authority.

Unpack that in steps, because it is the mechanism behind the first finding in the opening scenario.

1. A holder of edit can create a Pod and set its service account to any service account in that namespace.
2. That Pod receives the service account’s credentials.
3. So the holder can act with every permission any workload identity in that namespace holds — including permissions granted cluster-wide by a ClusterRoleBinding.
4. If a controller, operator or CI agent lives in that namespace with a ClusterRole that reads Secrets across the cluster, then edit in that one namespace is effectively that ClusterRole.

The indirect access path combines permission to run a Pod as a service account with that account’s actual bindings. Reading a Secret can also expose credentials where they are stored there, which is why the built-in view role excludes Secrets. This is also a secrets management concern. Not every Secret is a service-account token, and naming a ClusterRole alone does not grant its permissions.

Built-in role	What it says	The part people miss
view	Read-only on most objects	Excludes Secrets on purpose, and “does not allow viewing roles or role bindings”
edit	Read/write on most objects	Grants the API access of any ServiceAccount in the namespace
admin	Namespace administration via RoleBinding	Includes “the ability to create roles and role bindings within the namespace,” but “does not allow write access to resource quota or to the namespace itself”
cluster-admin	“Super-user access to perform any action on any resource”	In a RoleBinding it gives “full control over every resource in the role binding’s namespace, including the namespace itself“

The admin carve-out is a deliberate design worth copying: a tenant can manage their own permissions but cannot raise their own quota. Delegation of authority and allocation of capacity are different decisions with different owners.

Wildcard permissions expand without a role edit when an API group gains a matching resource, subresource, or verb. An all-resources/all-verbs role therefore acquires rights introduced by a later upgrade. Review that expansion as part of least privilege maintenance.

The RBAC API also constrains privilege escalation through role changes. To create or update a role, the caller must already hold its permissions at the same scope unless explicitly authorized for escalate. These API checks apply even without the RBAC authorizer. A legitimate delegation can consequently fail when the applying identity lacks the permissions it would grant.

Aggregated ClusterRole rules belong to the aggregation controller. With server-side apply, including rules in a manifest—even an empty list—claims that field. Applying can then conflict on .rules, or repeatedly clear the rules if conflicts are forced; the controller restores them. GitOps controllers that force conflicts can sustain this cycle. Omit the controller-managed field.

The operational conclusion is about the shape of the review, not its frequency. Auditing permissions means computing a union across three layers: the bindings, the roles they point at, and the identities those roles let you become. A list of RoleBindings answers none of it. If you cannot expand the third layer, you do not know what you granted.

A network policy you cannot tell is working

Kubernetes networking normally permits Pod-to-Pod communication without providing traffic encryption. A network policy selects Pods by label and constrains connections at the IP and port level; encryption needs a separate mechanism.

NetworkPolicy requires a network plugin or controller that implements its rules. API acceptance without that implementation provides no packet restriction; confirm the deployed CNI’s support before relying on policy objects.

A NetworkPolicy object can be accepted without an enforcing plugin. Its standard API does not report packet accept/drop events, but the plugin may expose state and logs. Validate both a working allowed path and an intended blocked path to a known listener. A failed connection alone could instead be DNS, routing, an absent server or another firewall. An audit log covers API activity, not proof that this packet policy was enforced.

Network isolation applies separately to each direction. Egress is unrestricted by NetworkPolicy until a selecting policy includes Egress in policyTypes. Isolation then limits connections to applicable allowances and documented exceptions; it is not a complete workload boundary.

From there, four consequences that determine how you debug.

• Order does not matter, and nothing conflicts. Applicable policies contribute allowances to a union for the selected Pod and direction. Evaluation order supplies no precedence, and another policy cannot subtract a connection already allowed by that union.
• Both ends must agree. The source’s egress allowance and destination’s ingress allowance must both permit a connection. Investigate both namespace policies when cross-team traffic is blocked.
• When policyTypes is omitted, Ingress is included and Egress is included if egress rules are present. An ingress-only policy does not change egress isolation. In the opening case, there were no egress rules and no other selecting egress policy; that combination left outbound traffic unrestricted by NetworkPolicy.
• Some edges are never closed. Standard NetworkPolicy cannot block a Pod’s access to itself or traffic between the Pod and its own node, regardless of their IP addresses. A deny-all guarantees denial only for TCP, UDP, and SCTP.

Start from default-deny where tenant separation requires it, then permit required DNS queries explicitly. Denying all egress otherwise blocks DNS as well. An empty namespaceSelector matches every namespace, so use a selector that expresses the intended namespace boundary.

Policy propagation creates two startup risks. A Pod may initially run before protection is installed, or become isolated before its allow rules arrive and temporarily lose connectivity. The Kubernetes API does not expose the exact transition. Design startup to tolerate changing connectivity; an init container can wait for a required destination rather than having the application exit at its first failed connection.

For established connections, the effect of a policy change is implementation-dependent. Deleting a Pod is also a request, not confirmation that its process has stopped: grace periods, unreachable nodes and controller replacement matter. During containment, prevent unintended recreation, verify termination on the node, and use a prevalidated network control when traffic must be blocked independently of that confirmation. Actual process exit ends that process’s communication; an API deletion response alone does not prove it.

Finally, what the API deliberately does not cover, as of the v1.37 documentation: no explicit deny, no targeting Services by name, no node-identity policies, no policies that apply across all namespaces by default, and no event logging. Those gaps are the reason service meshes and third-party policy engines exist, and the reason a policy layer is something you assemble rather than enable. Expressing the assembled rules as reviewed, versioned files — policy as code — is what keeps the union inspectable.

Quotas enforce admission limits, not reserved capacity

ResourceQuota limits covered resource requests or limits and object counts within a namespace. The admission mechanism rejects requests that would exceed the applicable quota; it is an enforced limit rather than a reporting total.

What it is not is a guarantee, and four documented properties explain why.

• Lowering quota does not evict existing workloads or reclaim their resources. Existing usage can remain above the new limit, and subsequent covered requests face that limit. Usage can fall through scaling down or deletion; restarting is not inherently required.
• Quota is independent of cluster capacity. Its absolute quantities do not grow when nodes are added. Administrators must review the namespace allocations separately from new physical capacity.
• The sum of namespace quotas can exceed cluster capacity. Passing quota admission therefore does not reserve a node or establish scheduling order. Pod priority, preemption and placement constraints also influence which pending work runs; arrival order alone does not describe the outcome.
• Quota does not dedicate nodes. Different namespaces can still contend on one node, and resource accounting is not a monetary bill cap. Node placement and runtime controls address additional noisy neighbor risks.

Compute quotas can require Pod resource declarations for the resources they track. LimitRange can supply defaults before quota evaluation, but those defaults still need to fit the workload and available quota. Verify the admitted Pod rather than assuming that adding a LimitRange makes every manifest acceptable.

Quota can reject Pod creation while accepting the Deployment that requests those Pods. The apply succeeds, but the ReplicaSet reports the rejection and the rollout stalls. Inspect that owner chain instead of looking only at the top-level request. The distinction follows from declared state being realized through a control loop.

Protect permission to update or delete ResourceQuota. A tenant that can raise its own limit defeats the allocation boundary; the built-in admin role consequently excludes quota writes.

Quota bounds covered resource consumption and reduces resource exhaustion, but it cannot isolate every shared resource, including network traffic. Stronger workload isolation may require node isolation and other runtime controls.

API Priority and Fairness addresses API-server contention. FlowSchema classifies requests by subjects and request attributes; PriorityLevelConfiguration controls concurrency allocation and queuing or rejection. A Pod’s scheduling priority does not itself give its API calls priority. A tenant controller can still create API load, so observe this layer separately from request-rate throttling and kernel resource controls.

What a workload may ask of the node

Requests and quotas govern how much. Pod Security governs what kind — specifically, which fields a Pod may set to reach past its own boundary. The Pod Security Standards define three levels that are “cumulative and range from highly-permissive to highly-restrictive.”

Level	Documentation’s description	Use
Privileged	“Unrestricted policy… This policy allows for known privilege escalations“	System and infrastructure workloads run by trusted operators
Baseline	“Minimally restrictive policy which prevents known privilege escalations. Allows the default (minimally specified) Pod configuration”	Ordinary application workloads
Restricted	“Heavily restricted policy, following current Pod hardening best practices”	Anything you would not personally vouch for

Baseline disallows host namespace sharing through spec.hostNetwork, spec.hostPID, and spec.hostIPC, privileged containers, and capabilities outside its permitted list. These settings can weaken the boundary between a container and its node. Validation covers the whole Pod: a violating listed container causes Pod validation to fail.

Namespace labels configure Pod Security enforce, audit and warn modes, with optional policy-version pins. Enforce rejects violating Pod requests within its scope; audit annotates audit events, and warn returns warnings. Choose levels and versions for each mode deliberately.

For workload resources such as Deployments, audit and warn evaluate the Pod template. Enforce acts on the resulting Pod requests rather than rejecting the Deployment template itself.

A Deployment apply can therefore succeed while its ReplicaSet cannot create new Pods. This is observable in FailedCreate Events and workload conditions; an update may keep old replicas running rather than produce an empty rollout. Warn can bring feedback to the author sooner, while audit supports review. Neither replaces checking rollout progress and the controller’s failure details.

Exemption scope determines which workload path avoids enforcement. Explicitly enumerate usernames, runtime class names, and namespaces, as the Pod Security documentation requires. Exempting an end user covers Pods they create directly, but not Pods created by a workload resource. Exempting a controller’s service account can instead implicitly exempt every user able to create the corresponding workload resource. That asymmetry explains why a person’s exemption may miss the intended path while a controller’s exemption can open a much broader one.

A per-mode version label can pin Pod Security to a Kubernetes minor’s policy; latest follows the current version. A floating policy can change admission outcomes at upgrade without a Pod edit. Pinning makes policy migration an explicit review task. Choose the behavior deliberately.

Observe pod_security_evaluations_total, pod_security_exemptions_total, and pod_security_errors_total on the API server. Non-fatal evaluation errors may fall back to the latest Restricted profile for enforcement; include that behavior in diagnosis.

Pod Security constrains what a workload requests but does not supply a separate kernel. Ordinary Linux containers share the host kernel and mount host interfaces such as /sys and /proc; hardware-virtualized VMs provide a different boundary. For untrusted or potentially malicious code, evaluate sandboxing with a VM or userspace kernel per Pod. A stricter namespace label does not remove a container-breakout vulnerability.

Node isolation dedicates nodes to each tenant to prevent tenant Pod co-location. Taints can combine with a mutating webhook that supplies the required tolerations and node affinities. Billing by dedicated node can be simpler than attributing individual sandboxed Pods. The kubelet and, without virtual control planes, the API service still remain shared. An attacker may use node or workload permissions to move laterally, so node dedication is not a complete cluster boundary.

DNS visibility and service connectivity are separate controls. Kubernetes DNS permits cross-namespace lookups by default. If a tenant can reach the DNS service, it may resolve known or guessed names in another namespace even when policy blocks connections to the target service. This does not by itself provide a complete list of service names: API listing permissions and mechanisms such as an explicitly enabled DNS zone transfer are separate discovery paths. CoreDNS can restrict which namespaces or records are exposed, with additional policy needed for the intended caller boundary. Storage also crosses namespace boundaries: a PersistentVolume is cluster-scoped and outlives workloads, while its PersistentVolumeClaim is namespaced.

A shared class with Delete reduces the risk of reusing a former tenant’s retained volume, but it may conflict with recovery requirements. Separate StorageClasses help divide provisioning and defaults; they do not alone enforce who may select a class or access its assets. Combine admitted class choices, actual PV reclaim policies, backend permissions and controlled rebinding or sanitization. Retention and prevention of cross-tenant reuse can coexist when those procedures preserve ownership boundaries.

How far a mistake travels

Blast radius asks how much a fault can affect. AWS’s cell example leaves 90 of 100 requests unaffected when one of ten cells fails. That arithmetic assumes equal request allocation, a failure contained in one cell and healthy shared dependencies. If the failed cell serves half the traffic, its request impact can be half the workload. The assumptions make the example useful; cell count alone does not determine impact.

Cluster-scoped objects deserve review because namespaces do not partition their definitions. CRDs, StorageClasses and webhook configurations can create shared dependencies, but their actual effect depends on what they control or match.

For an admission webhook, inspect rules, namespace and object selectors, match conditions, timeout and failurePolicy. A broadly matched fail-closed webhook can block deployments across tenants when it is unavailable; a narrowly matched one has a different reach. Ignore applies to calling errors, not to a successful webhook response that explicitly denies the request. The cluster scope of the configuration alone does not mean every API request invokes it.

Which gives a decision that is about degree of separation rather than about tools.

Level	What it contains	What it costs
Namespace per tenant	API resources, most policies. “Negligible resource cost”	“Can be difficult to configure,” and leaves CRDs, StorageClasses and webhooks shared
Virtual control plane per tenant	Separates tenant API objects and many policy conflicts; host and synchronization dependencies remain	Operating each control plane and its sync layer, plus separate data-plane isolation
Cluster per tenant	Nearly everything, up to “dedicated hardware if VMs are not considered an adequate security boundary”	“The cost and complexity of managing multiple clusters”

A virtual control plane can resolve many problems caused by sharing a tenant-facing API server. Review which objects are isolated and which host-cluster, synchronization and management services remain shared. Worker-node security and contention still need separate controls. The resulting boundary depends on the implementation, not only on assigning each tenant an API endpoint.

Contain change as well as failure. A ship’s bulkheads divide its interior into watertight compartments; the same idea motivates a canary deployment and a tested rollback. Upgrading every node together instead creates a cluster-wide change radius. Apply containment to the platform’s own upgrade procedure.

A small observed incident does not prove a small worst-case radius. It can still provide evidence about dependencies and failure propagation. Record actual affected requests or tenants separately from the architecture’s potential scope, then test assumptions about shared objects and partial failure.

The cluster you secured is not the cluster you will run

Everything above describes a configuration. Kubernetes does not let a configuration sit still, and the constraints are published rather than discovered, so this is plannable work rather than a surprise.

Kubernetes patch support has phases. The project describes roughly twelve months of standard support followed by two months of maintenance, during which releases may address assigned CVEs, dependency issues and critical core-component issues. Maintenance narrows eligible fixes; it does not exclude every bug fix or guarantee a patch for every eligible issue. Consult the branch’s dates and advisories when planning the support lifecycle.

Versions use x.y.z and Semantic Versioning terminology. Each component has its own supported version skew relative to the API server.

Component	Allowed relative to kube-apiserver
kube-apiserver (HA peers)	Newest and oldest “must be within one minor version“
kubelet	“Must not be newer”; “may be up to three minor versions older“
kube-proxy	Same as kubelet, and within three minors of the kubelet beside it
controller-manager, scheduler, cloud-controller-manager	“Must not be newer”; “may be up to one minor version older (to allow live upgrades)”
kubectl	“Within one minor version (older or newer)”

Upgrade ordering follows the supported combinations: API servers first, then the controller manager, scheduler and cloud controller manager; node components can follow while remaining within their allowed skew. kube-apiserver upgrades must not skip minor versions. Deployment tools and providers may impose additional requirements.

Four consequences matter more than the table itself.

• A kubelet already three minors behind blocks advancing the API server to the next minor until the node catches up. That does not prohibit a patch within the current control-plane minor. Check which branches contain the required security fix: node catch-up becomes part of that fix’s critical path when it requires a newer minor.
• Before a minor kubelet upgrade, drain workloads as required by the supported procedure; in-place minor kubelet upgrades are unsupported. User-visible continuity also depends on replica placement, spare capacity, disruption budgets and application graceful shutdown. Correct signal handling alone cannot guarantee availability.
• A mixed control plane narrows supported combinations. While HA API servers run different minors, the supported kubelet and kubectl ranges must satisfy those simultaneous constraints. Check operator tooling as well as nodes during the intermediate state.
• Your policy layer is a prerequisite. Registered admission webhooks must handle the resource versions and fields sent by the new API server. matchPolicy: Equivalent can help match equivalent resource versions. Validate the isolation webhook before advancing the upgrade.

The documented preparation is to reach the latest patch of the current minor before moving to the target minor’s latest patch. Validate the provider or deployment tool’s procedure and prerequisites as part of that preparation.

APIs leave on a schedule

Version skew tells you how far apart components may be. The deprecation policy tells you what disappears while you wait, and it is unusually specific about time — which makes it possible to plan against instead of reacting to.

An API group version cannot lose an existing element or substantially change its behavior in place; removal requires a new group version. GA API versions may be deprecated but remain within the same Kubernetes major version. Beta versions are deprecated no later than the longer of nine months or three minor releases after introduction, then stop being served after the longer of nine months or three minors following deprecation. Alpha versions may disappear in any release without prior notice.

API lifetime and supported component combinations are separate contracts. The deprecation page’s rationale still refers to a maximum skew of two releases; do not substitute that phrase for the current component-specific Version Skew Policy, whose kubelet allowance can be three minors. Both policies matter for avoiding a breaking change, but they answer different compatibility questions.

Three other clocks catch people out, and they are easy to miss because each belongs to a different team.

• Admin-facing flags have a shorter runway. Deprecated GA elements of user-facing CLIs such as kubectl remain usable for the longer of twelve months or two releases. Admin-facing components such as kubelet have the longer of six months or one release. The nominal minimum is half as long; compare the actual release schedule too. Deprecated behaviors have a separate minimum of one year.
• Feature gates default differently by stage. Alpha gates are disabled unless enabled; beta gates are enabled unless disabled. Gate retirement has its own schedule: beta-to-GA allows six months or two releases, beta-to-EOL allows three months or one release, and alpha-to-EOL can occur immediately.
• A deprecated stable metric has a documented migration window before it becomes hidden. An ordinary threshold alert can return no series and stop firing when its metric disappears, but that is not inevitable: absent or absent_over_time checks and the monitoring system’s no-data handling can detect the gap. Test rules against missing data. Temporarily exposing hidden metrics can assist migration, not replace it.

Maintain inventories of API versions, feature gates, metrics and alert rules, and admission-webhook compatibility. Include operators and controllers as API consumers. Validate both normal and missing-metric alert behavior before upgrades, and assign an owner to deprecation migrations before their support windows close.

What the platform team owes

Pull it together into what someone actually has to do, because “we use namespaces and RBAC” is not an answer to any of the above.

Tenant onboarding combines namespace policy, effective identity permissions, protected quotas, appropriate resource defaults, versioned Pod Security and tested network allowances. Storage needs admitted class selection, PV retention and access controls as well as a chosen default. Record the owner and observable outcome for each control.

Verify the mechanism and its outcome. API acceptance, an admission rejection, a working route and a successful rollout are different observations; collect the evidence that matches each claim.

• Network: test a known listening destination from an allowed source and a source that should be blocked. Match direction, protocol, port, namespace and selectors, account for node-local exceptions, and inspect plugin evidence. Re-test relevant cases after network changes.
• Admission: in a controlled test namespace, compare an allowed Pod with a deliberate policy violation. For a violating Deployment template, check warn feedback where configured and inspect ReplicaSet FailedCreate Events and Deployment conditions. A successful apply alone is not completion.
• Quota and defaults: inspect the admitted resource values for a Pod with omitted requests, then separately test a request that exceeds the configured quota. Interpret defaults, rejection and scheduler placement as different results.
• A permission review that expands bindings into roles into reachable identities, rather than listing bindings.
• pod_security_exemptions_total and the audit log as standing signals, since exemptions and violations are the things that accumulate quietly.

Assign policy decisions jointly to the owners of capacity, budgets and platform operation. Record an owner and review or expiry date for exemptions. Negotiate upgrade windows against service objectives and the actual branch lifecycle, including the narrower security, dependency and critical-issue coverage in maintenance.

A requirement for stronger separation may justify a virtual control plane or a separate cluster. Verify their remaining shared dependencies and data-plane controls rather than assuming the label establishes the boundary. Untrusted code may require a sandboxed execution environment as well as namespace policy. Keep upgrade inventories current enough to act before compatibility or support becomes the constraint.

In the opening case, Pod creation rights exposed a broadly authorized service-account identity; ingress-only policy left egress unrestricted because no other egress policy selected those Pods; and node skew prevented the next minor upgrade. Those conditions identify what to repair. Effective permissions, enforced packet rules and supported upgrade paths must each be checked against the actual configuration.

References

• Kubernetes: Multi-tenancy
• Using RBAC Authorization
• Network Policies
• Resource Quotas
• Pod Security Standards
• Pod Security Admission
• Version Skew Policy
• Patch Releases
• Kubernetes Deprecation Policy
• AWS Well-Architected: Reducing the Scope of Impact with Cell-Based Architecture
• Kubernetes: API Priority and Fairness
• Kubernetes: Dynamic Admission Control
• Kubernetes: Pod Priority and Preemption
• Kubernetes: Pod Lifecycle
• Prometheus: absent and absent_over_time

CoreDNS: kubernetes plugin — namespace exposure and zone transfers