Containers: Images, Isolation, and Limits

A container is easy to run and hard to describe precisely. The command that starts one takes a few words, and the result behaves like a small machine: it has its own filesystem, its own process list, its own network address, and a memory limit. None of those properties comes from a single mechanism. A container is an ordinary Linux process (or a group of processes) whose environment has been assembled from several independent kernel features and a packaged filesystem. When something goes wrong, such as a process killed without an error, a service that cannot reach another one, or a host that slows down because of a neighbor, the cause is almost always in one of those separate parts. The skill this post builds is knowing which part to inspect.

What this post assumes. From Deploying the Same Artifact Across Environments it assumes you know why the artifact that was tested should be the artifact that is promoted, and why environment-specific configuration should stay outside it. This post explains what that artifact is when it is a container image, and what the runtime adds when it runs. From general computer science it assumes familiarity with processes, virtual memory, file systems, system calls, and hashing. Linux-specific concepts (namespaces, control groups, capabilities, overlay file systems) are introduced here. The previous post, Software Supply Chain Controls in Delivery Pipelines, covered how images are verified and signed. This post explains what is inside them.

What you should be able to do afterward. Describe the structure of an image and predict what a rebuild will change; explain what each namespace type separates and what it does not; set and verify memory, CPU, and process limits and interpret the symptoms when a limit is reached; decide which mounts, capabilities, and user settings a workload needs; recognize where the isolation boundary ends; and follow an evidence-based sequence to diagnose a container problem.

How the examples were produced. The image-layer example below is a Python program written for this post and run locally on macOS (Python 3.12, standard library only), and its output is real. The CPU-quota cases are short hand calculations. Namespaces and cgroups are Linux kernel features, so the commands that inspect them were not executed for this post: no Linux host or running container daemon was used. Those commands are labeled as not executed, and the behavior they describe comes from the Linux manual pages, the kernel documentation, the OCI specifications, and the Docker documentation cited in the references.

The model: a process, a filesystem, views, and limits

Four statements define an ordinary Linux container. Each statement corresponds to a different mechanism, and each can be configured separately.

• The filesystem comes from an image. A container image is a packaged root filesystem plus defaults for how to start it. This is what you build, scan, sign, and promote.
• The views come from namespaces. A Linux namespace gives a set of processes its own view of one kind of system resource, such as process IDs, mount points, or network interfaces.
• The limits come from control groups. Control groups (cgroups) account for resource use by groups of processes and can cap it.
• The privileges come from several settings. The user the process runs as, its Linux capabilities, a system-call filter, and mandatory access control policy together decide what the process may ask the kernel to do.

The kernel is shared. Unlike a virtual machine, which runs its own guest kernel behind a virtualization boundary, an ordinary container runs on the host’s kernel. This single fact explains most of the security discussion later in the post: the container boundary is made of rules enforced by a kernel that every container on the host also uses. The Docker documentation on security states that containers share the host kernel and describes namespaces and cgroups as the two foundations of Docker’s isolation. The container glossary entry summarizes the same arrangement and contrasts it with runtimes that use virtual machines; the OCI container standards are about packaging and lifecycle interfaces and do not fix the strength of isolation.

The following text sketch summarizes the layers. It is a mental model drawn for this post, not a representation of any specific runtime’s internals.

  image (read-only layers)      -> what files exist, default user, default command
  + writable layer (per container)
  + mounts (volumes, bind mounts, tmpfs)
  ------------------------------------------------------------
  one or more processes, run with:
    namespaces   -> which resources the processes can see (pid, mnt, net, ipc, uts, user, cgroup, time)
    cgroups      -> how much CPU, memory, processes, I/O they may consume
    capabilities, seccomp, MAC policy, uid/gid -> which kernel operations they may attempt
  ------------------------------------------------------------
  shared host kernel

Image layers

What an image is made of

The OCI image specification defines an image as a set of content-addressed pieces. A manifest lists a configuration and an ordered list of layers, each referenced by a descriptor that carries the content’s digest and size. The manifest requires the base layer at index 0, and the final filesystem must equal the result of applying all layers in order to an empty directory. A configuration holds execution defaults such as the user, the command, and environment variables, and the specification says plainly that these are defaults that can be overridden when the container is created. An optional index lists several manifests, usually one per platform, so that a client can select the one that matches its operating system and CPU architecture. Because each reference carries a digest, the whole structure forms a hash tree: changing any layer changes the manifest, and changing the manifest changes the digest used to name the image. That is the property the previous post relied on when it pinned images and signed digests.

The container image entry in this series’ glossary describes the same four parts. For the relationship between an image and a running instance, see container image and container: an image is a template, a container is an instance created from it with its own writable layer and runtime configuration.

A layer is a changeset

An OCI layer is not a snapshot of a whole filesystem. It is a changeset, serialized as a tar archive, that records additions, modifications, and removals relative to the layers below it. The layer specification says layers are applied, not merely extracted: applying a layer to an existing path replaces it, and a deletion is expressed by a special entry named with the prefix .wh. followed by the base name of the path to delete. A directory can also be hidden wholesale with an opaque whiteout.

The consequence is easy to state and often missed. A deletion in a later layer hides a file; it does not remove the bytes from the earlier layer. Anyone who can pull the image can read the earlier layer’s tar archive. This is why copying a credential into an image and deleting it in a later step does not remove it. The following program builds a two-layer image by hand, with a placeholder instead of a real secret, applies the layers in order, and then inspects the first layer directly. The configuration lists the digest of each uncompressed layer under rootfs.diff_ids, which is the field the image configuration specification defines for that purpose. Layers here are uncompressed, so those values equal the layer digests in the manifest.

"""Build a tiny OCI-style image by hand: two layers, one deleting a file with a whiteout.
Everything here is invented demo data (the 'secret' is a placeholder string)."""
import hashlib, io, json, tarfile

def sha(b): return "sha256:" + hashlib.sha256(b).hexdigest()

def make_layer(entries):
    buf = io.BytesIO()
    with tarfile.open(fileobj=buf, mode="w", format=tarfile.PAX_FORMAT) as t:
        for name, data in entries.items():
            ti = tarfile.TarInfo(name); ti.size = len(data); ti.mtime = 0; ti.uid = ti.gid = 0
            t.addfile(ti, io.BytesIO(data))
    return buf.getvalue()

layer1 = make_layer({"app/server": b"#!/bin/sh\necho serving\n", "app/.env": b"API_TOKEN=PLACEHOLDER-NOT-A-REAL-SECRET\n"})
layer2 = make_layer({"app/.wh..env": b"", "app/config.yaml": b"port: 8080\n"})   # whiteout = deletion record

config = json.dumps({"architecture": "amd64", "os": "linux",
                     "config": {"User": "1000", "Cmd": ["/app/server"]},
                     "rootfs": {"type": "layers", "diff_ids": [sha(layer1), sha(layer2)]}}, sort_keys=True).encode()
manifest = json.dumps({"schemaVersion": 2, "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {"mediaType": "application/vnd.oci.image.config.v1+json", "digest": sha(config), "size": len(config)},
    "layers": [{"mediaType": "application/vnd.oci.image.layer.v1.tar", "digest": sha(l), "size": len(l)} for l in (layer1, layer2)]},
    sort_keys=True).encode()

def apply(layers):
    fs = {}
    for blob in layers:                       # base layer first
        with tarfile.open(fileobj=io.BytesIO(blob)) as t:
            for m in t.getmembers():
                d, base = m.name.rsplit("/", 1)
                if base.startswith(".wh."):
                    fs.pop(d + "/" + base[4:], None)      # hide the lower path
                else:
                    fs[m.name] = t.extractfile(m).read()
    return fs

print("manifest digest :", sha(manifest))
print("config digest   :", sha(config))
for i, l in enumerate((layer1, layer2)): print(f"layer {i} digest  :", sha(l), len(l), "bytes")
fs = apply([layer1, layer2])
print("merged filesystem:", sorted(fs))
print(".env visible to the running container?", "app/.env" in fs)
with tarfile.open(fileobj=io.BytesIO(layer1)) as t:
    print("still readable in layer 0:", t.extractfile("app/.env").read().decode().strip())
print("layer 1 entries:", [m.name for m in tarfile.open(fileobj=io.BytesIO(layer2)).getmembers()])
# change one byte of the base layer and see every dependent digest change
layer1b = layer1.replace(b"serving", b"servinG")
print("base layer digest after a 1-byte change:", sha(layer1b)[:19], "(was", sha(layer1)[:19] + ")")

Output:

manifest digest : sha256:2d43a36b8c6dcda8c83bcf8352d3584016e3658ed37edccdfd6494d7ee8fc457
config digest   : sha256:29eff073c2efbf7fe2ce896d19055d19b58f13cf99b62c0774d25b6f3bbc66dd
layer 0 digest  : sha256:4e895ece65ebcfad1421c986ae8789870f59ff74bc064b918147a91b55f1de1e 10240 bytes
layer 1 digest  : sha256:fafd0ec2fea393d8569b20ba5c69ae33cff9bde9c8b4605360263dc57be403cc 10240 bytes
merged filesystem: ['app/config.yaml', 'app/server']
.env visible to the running container? False
still readable in layer 0: API_TOKEN=PLACEHOLDER-NOT-A-REAL-SECRET
layer 1 entries: ['app/.wh..env', 'app/config.yaml']
base layer digest after a 1-byte change: sha256:cfe6bde04f3c (was sha256:4e895ece65eb)

Four observations follow from this run. First, the merged view, which is what the container’s process would see, does not contain app/.env. Second, layer 0 still contains it, and the second layer’s only trace of the deletion is the whiteout entry app/.wh..env. Third, the 10240-byte size of each layer is an artifact of tar block padding in this tiny example and says nothing about real images. Fourth, changing one byte in the base layer changes that layer’s digest, and because the manifest and configuration refer to layer digests, the identity of the whole image changes with it. The digests above come from this demonstration and are not output from any real image.

Treat a secret that reached a distributed layer as disclosed. Revoke or rotate it, and fix the build so that secrets are supplied through a build mechanism that does not write them into a layer. Docker’s build documentation says that build arguments and environment variables are inappropriate for secrets because they persist in the final image, and that a secret mount is available only for the duration of the instruction that uses it. The glossary entry on container images notes that this applies to data retained in the final image’s layers or configuration, not to every file used during a build: a multi-stage build whose earlier stage is not part of the final image does not distribute that stage’s layers, and build secret mounts avoid recording the secret bytes. Logs and exported caches are still places where a secret can persist, so verify them separately.

From layers to a running filesystem

At run time, the Docker documentation describes how the overlayfs storage driver combines the image layers and one writable layer. The image layers are mounted read-only as the lower directories, a new directory for the container becomes the upper directory, and the container sees a merged directory. When a container first modifies a file that exists in an image layer, the driver copies the entire file up into the writable layer, and later writes go to that copy. Deleting a file creates a whiteout in the writable layer, and deleting a directory creates an opaque directory. The image layers are never modified.

Three operational consequences follow. The writable layer is per container and is removed when the container is removed, so state that must outlive a replacement belongs in a volume or an external store, not in the container filesystem. Two containers started from one image do not share their writable layers, and a change made inside a running container does not update the image or any sibling; to distribute a change, build a new image. Finally, a workload that writes heavily into the container filesystem pays the copy-up cost for each file it first modifies, and its data lives in a layer whose lifecycle is the container’s. Put write-heavy paths on a volume or a memory-backed mount designed for the purpose.

Layers, caches, and rebuilds

Because layers are content-addressed, a registry or a node can reuse a blob it already holds, so two images that share a base can share the stored base layers. Docker’s builder cache works at the instruction level: when an instruction differs from its cached counterpart, the cache is invalidated for the instructions after it. The documentation advises ordering instructions from least to most frequently changing, which shortens rebuilds. Two caveats keep this honest. A rebuilt step may produce a layer with a different digest even if you did not intend a change, for example because it embedded a timestamp, and that matters for reproducible builds and for transfer. And a cache can preserve old packages: the same documentation points out that a cached package-install instruction is not invalidated automatically between builds, so a rebuild a week later can reuse the old layer unless you force re-execution. Keeping an image patched means rebuilding from updated inputs and promoting a new digest, as the earlier supply chain post described; it does not mean updating files inside a running container.

Defaults in the image and the runtime override

The image configuration carries a default user. Setting it to a non-root numeric ID makes non-root the default for containers started from the image, and the specification states that this is a default that can be overridden at creation. If the runtime is told to run as user 0, the image default has no effect. Review the image’s defaults and the runtime’s effective settings together, because the second can silently override the first. For image names and digests, the image tag and digest entry and the previous post cover why a deployment should reference a digest and record which platform manifest was executed.

Processes and namespaces

What a namespace separates

The Linux manual page on namespaces lists eight types. Each one wraps a single kind of global resource so that processes inside the namespace see their own instance of it.

Namespace	Separates	What it does not do
PID	Process IDs	Processes in an ancestor namespace can still see and signal descendants, subject to permission checks
Mount	The list of mount points	Does not limit the files reachable through mounts you create; a bind mount of a host directory exposes that directory
Network	Network devices, protocol stacks, ports, routing and firewall rules	Does not decide what the namespace can reach once it has an interface attached to something
User	User and group IDs, and capabilities within the namespace	Mapping a user does not remove capabilities or policy; privileges depend on more than the ID
IPC	System V IPC objects and POSIX message queues	Shared memory between containers is still possible if namespaces are shared
UTS	Hostname and NIS domain name	Mostly cosmetic
Cgroup	The root directory a process sees for cgroups	Does not limit anything; it hides where the process sits in the hierarchy
Time	Boot and monotonic clocks	Does not change the real-time clock

A namespace isolates one named kind of thing. There is no single “container namespace”. The creation APIs are clone(2) and unshare(2), which place a process in a new namespace, and setns(2), which joins an existing one by file descriptor. Each process has a link per namespace type in /proc/[pid]/ns/, and two processes are in the same namespace when the targets of those links are identical. A namespace is torn down when its last process leaves, unless something keeps it alive, such as an open file descriptor or a bind mount of the link.

A Kubernetes namespace is a different concept. It groups API objects for naming, quota, and policy. It does not create a Linux namespace and does not by itself isolate processes. Because the word is reused, say “Linux namespace” or “Kubernetes namespace” explicitly in design documents.

PID namespaces and the container’s first process

In a new PID namespace, the first process becomes PID 1. The pid_namespaces manual page records three behaviors that matter in production. Signals sent to PID 1 from inside its own namespace are delivered only if PID 1 has installed a handler for them, with SIGKILL and SIGSTOP from ancestor namespaces as the exceptions. When PID 1 terminates, the kernel kills every other process in the namespace. And orphaned processes are reparented to the namespace’s init.

The practical consequence is that the process you configure as a container’s command is an init process whether you intended it or not. If it does not install a handler for the termination signal, a polite stop request can be ignored until the runtime escalates to SIGKILL. If it spawns children and does not wait for them, finished children remain as zombies. If it exits, everything else in the container dies with it. The container’s process, its signal handling, and its exit status are the first things to check when a container does not stop cleanly. The Docker documentation lists exit codes 125, 126, and 127 as errors in starting the container command (daemon error, command cannot be invoked, command not found), and any other code as the command’s own exit code. By shell convention an exit code above 128 means the process was ended by signal number code minus 128, so 137 corresponds to signal 9, SIGKILL. That convention is not Docker-specific and does not by itself say who sent the signal; the memory-limit section shows how to distinguish an out-of-memory kill from a manual one.

Mount namespaces and the root filesystem

A new mount namespace starts with a copy of its parent’s mount list, and later mounts and unmounts in one namespace do not affect the other unless they are in a shared peer group. The manual page describes four propagation types: shared, private, slave, and unbindable. The OCI runtime specification exposes the same idea through a rootfsPropagation field, which can be shared, slave, private, or unbindable. A container runtime builds the root filesystem from the image layers, mounts a few special filesystems, and switches the process’s root to it with chroot or pivot_root.

Two mount-related rules matter for platform engineers. A mount hides what was already in its destination directory: the Docker documentation on bind mounts notes that pre-existing files at the target are obscured by the mount, which explains “my files are missing” reports after adding a volume over an application directory. And mounts are an escape hatch from the image filesystem: a bind mount is writable by default, so a process in the container can create, modify, or delete files on the host path, and the documentation recommends read-only mounts where writes are not needed. The bind mount and volume entry covers the differences in ownership and lifecycle. A named volume is managed by the container engine and survives container removal, while a memory-backed mount disappears when the container stops.

Network namespaces and published ports

A network namespace has its own interfaces, addresses, routing table, firewall rules, and ports. A container started with its own network namespace therefore sees an empty network until the runtime connects it to something. In Docker’s bridge networking, each container gets its own network namespace and an interface with an address from the bridge’s subnet. Containers attached to the same bridge can reach each other, and containers on different bridges cannot, except through published ports or routing at the host. On a user-defined bridge, containers resolve each other by name, while the default bridge resolves only by IP address.

Publishing a port with -p makes the container’s port reachable from outside the host on the host’s addresses by default, as the Docker documentation states. This is a boundary crossing, not a convenience flag. Treat each published port as an entry in the service’s attack surface, bind it to a specific host address when it should not be reachable from everywhere, and verify reachability from the places that should and should not have it.

User namespaces and what root means

Inside a container, a process can be UID 0 without being root on the host, but only if a user namespace maps container UID 0 to an unprivileged host UID. Docker’s userns-remap feature does this: it maps the container’s root to an unprivileged range of host IDs defined in /etc/subuid and /etc/subgid, so that a process that escapes the namespace holds only an unprivileged host identity. Rootless mode goes further by running the daemon itself without root. The documentation lists restrictions, including incompatibility with sharing the host’s PID or network namespace and limits on some storage drivers and privileged mode.

Without a user namespace mapping, UID 0 in the container is UID 0 on the host as far as file ownership and many kernel checks are concerned, and what limits it is the reduced capability set, the system-call filter, and policy modules. That is a thinner boundary than a mapped identity. Which configuration your platform uses is a security-relevant fact to document.

Commands that show namespace boundaries (not executed)

The following commands show how these boundaries are observed and created on a Linux host. They were not executed for this post. Their documented behavior is taken from the unshare(1), nsenter(1), and namespaces(7) manual pages.

# Not executed in this post.

# Compare namespace identities of two processes: same link target means same namespace.
ls -l /proc/self/ns/ /proc/<container-process-pid>/ns/

# Create a new PID namespace with a fresh /proc; --fork makes the child PID 1.
sudo unshare --fork --pid --mount-proc sh -c 'echo "my pid: $$"; ps -e'

# Create a user namespace mapping the current user to root inside it (no host root needed).
unshare --user --map-root-user sh -c 'id; cat /proc/self/uid_map'

# Join an existing container process's network namespace and look at its interfaces.
sudo nsenter --target <container-process-pid> --net ip addr

What each command shows, according to the documentation: in the PID namespace, the shell reports a small PID such as 1 and the process list contains only the namespace’s processes. In the user namespace, id reports UID 0 while uid_map shows the mapping to an unprivileged host ID. In the nsenter case, the interfaces listed are the container’s addresses instead of the host’s. These descriptions are derived from the documentation, not from a run. nsenter into a container’s namespaces is a powerful diagnostic and also a boundary-crossing operation, so it should be restricted to people authorized to administer the host.

Control groups and resource boundaries

What cgroups do

Control groups organize processes into a hierarchy and attach controllers to it. A controller accounts for and can limit one kind of resource: the memory controller for memory, the CPU controller for processor time, the pids controller for the number of processes, and an I/O controller for block I/O. The cgroups manual page describes two versions. Version 1 lets each controller have its own hierarchy, which the page says led to inconsistencies. Version 2 (Linux 4.5 and later) uses a single unified hierarchy, and a controller cannot be used in both versions at once. The interface is a pseudo-filesystem, usually mounted at /sys/fs/cgroup, where each cgroup is a directory whose files are the controls. In version 2, a non-root cgroup that has child cgroups does not directly contain processes; processes live in leaf cgroups, which is the “no internal processes” rule.

Namespaces and cgroups are independent. A PID namespace says nothing about memory. A process can be in a fresh PID namespace with no memory limit, or in a memory-limited cgroup with the host’s PID namespace. The Docker documentation states the default plainly: containers have no resource constraints and can use as much of a resource as the host kernel scheduler allows. A container started without limits is not bounded, however isolated its views are. The noisy neighbor entry describes the contention this allows, and the resource exhaustion entry describes the failure class.

Memory

The kernel documentation for cgroup v2 distinguishes two memory thresholds that are easy to confuse. memory.max is the hard limit: when usage reaches it and memory cannot be reclaimed, the OOM killer is invoked within the cgroup. memory.high is a throttling threshold: processes that exceed it face heavy reclaim pressure, but the OOM killer is not invoked by it. memory.current reports current usage including descendants. memory.events has counters including oom (how many times the limit was about to be exceeded) and oom_kill (how many processes were killed). The documentation also notes that some kinds of allocation do not lead to an OOM kill and instead fail or return an error to user space.

The Docker flag --memory sets a hard limit, and --memory-swap sets the combined memory and swap total, so setting the two equal gives no swap. Docker’s documentation also warns against disabling the OOM killer on a container that has no memory limit, because the host can become unstable.

These facts define a diagnostic sequence for a container that “just died”. An exit code of 137 suggests SIGKILL; confirm whether the memory cgroup’s oom_kill counter increased, whether the container runtime recorded an out-of-memory kill in its state, and what the kernel logged. Without that evidence, a 137 could equally be a manual kill or a runtime forced stop after a grace period. A memory limit set too close to a service’s real working set produces kills under load that are invisible in functional tests, so set limits from measurement of peak usage under realistic load, include headroom, and alert on memory.events instead of waiting for restarts.

CPU

The CPU controller offers two different mechanisms. cpu.weight (range 1 to 10000, default 100) divides CPU proportionally among sibling cgroups when they compete; it does not cap anything and has no effect when the CPU is idle. cpu.max has the form $MAX $PERIOD and means the group may run for $MAX microseconds in each $PERIOD; the value max means no limit. When a group uses up its quota within a period, it is throttled until the next period, and cpu.stat counts the periods in which that happened as nr_throttled. Docker’s --cpus flag expresses a limit in CPUs, such as 1.5, and --cpu-shares is a relative weight (default 1024) that has an effect only under contention.

The quota is a sum over all threads in the cgroup, and it is enforced per period, so a limit does not mean slower execution at a constant rate; it means execution until the quota is spent and then a pause. That has two consequences. A multi-threaded service can exhaust a quota early in a period and then stall for the rest of the period, which shows up as tail latency rather than as high average CPU. And a bursty workload with a low average can be throttled even though its average is under the limit. Two short calculations show it, assuming a period of 100,000 microseconds (100 ms) and invented demand. They are arithmetic on the documented rule, not measurements of any kernel.

Case	Setting	Demand	Result under the rule
1. Four busy threads	1.5 CPUs, so cpu.max of 150000 100000	Each thread wants 100 ms of CPU per period, 400 ms in total	At most 150 ms of CPU runs per period, so the group is throttled in every period and 250 ms of wanted CPU time is left unfinished each period
2. Bursty service	1.0 CPU, so cpu.max of 100000 100000	Averages 0.5 CPU, but one period needs 200 ms	Only 100 ms runs in that period, so nr_throttled rises by one even though the long-run average is half the limit

A short Python model of the cpu.max rule reproduces the same arithmetic and extends it over several periods. It simulates the documented rule and does not measure any kernel. Its output is from a local run.

"""Model of the cpu.max semantics '$MAX $PERIOD' (microseconds). A simulation of the rule, not a measurement."""
def run(cpus, period_us, demand_ms_per_window, windows=5):
    quota_us = int(cpus * period_us)
    print(f"cpu.max would read: {quota_us} {period_us}   (limit = {quota_us/period_us:.2f} CPUs)")
    remaining_work = 0
    throttled = 0
    for w in range(windows):
        remaining_work += demand_ms_per_window * 1000          # microseconds of CPU the threads want to run now
        run_us = min(remaining_work, quota_us)                 # at most quota per period, summed over all threads
        remaining_work -= run_us
        if remaining_work > 0: throttled += 1                  # quota exhausted before the period ended
        print(f" window {w}: ran {run_us/1000:.0f} ms of CPU, backlog {remaining_work/1000:.0f} ms, nr_throttled so far {throttled}")
print("Case 1: 4 busy threads on a 1.5-CPU limit, 100 ms period (each wants 100 ms per window = 400 ms CPU)")
run(1.5, 100000, 400, windows=3)
print("Case 2: bursty service averaging 0.5 CPU but wanting 200 ms in one window on a 1.0-CPU limit")
run(1.0, 100000, 200, windows=1)

Output:

Case 1: 4 busy threads on a 1.5-CPU limit, 100 ms period (each wants 100 ms per window = 400 ms CPU)
cpu.max would read: 150000 100000   (limit = 1.50 CPUs)
 window 0: ran 150 ms of CPU, backlog 250 ms, nr_throttled so far 1
 window 1: ran 150 ms of CPU, backlog 500 ms, nr_throttled so far 2
 window 2: ran 150 ms of CPU, backlog 750 ms, nr_throttled so far 3
Case 2: bursty service averaging 0.5 CPU but wanting 200 ms in one window on a 1.0-CPU limit
cpu.max would read: 100000 100000   (limit = 1.00 CPUs)
 window 0: ran 100 ms of CPU, backlog 100 ms, nr_throttled so far 1

Read the calculations for what they show and what they simplify. In case 1, demand exceeds the cap in every period, so unfinished work accumulates; in a real service that backlog appears as queued requests and latency, and the arithmetic ignores scheduling details. In case 2 a single burst larger than the quota is cut off within its period although the average is fine. The practical conclusion is to watch the throttling counter and latency percentiles, not only average CPU utilization, and to size CPU limits for burst behavior or use requests and weights without a hard cap where the platform supports it. Whether to cap at all is a trade-off: a cap protects neighbors and makes capacity predictable, and it can add latency to the capped workload.

Processes, I/O, and what limits do not cover

pids.max is a hard ceiling on the number of processes in a cgroup. The kernel documentation says a cgroup’s PID policy cannot be violated through fork() or clone(); an attempt beyond the limit fails with EAGAIN. Docker exposes it as --pids-limit. This is the control that contains a fork bomb. An application that creates a thread per request will start returning errors when it reaches the ceiling, so size it for the workload, not at an arbitrary small number.

Not everything is limited by these controllers. Network bandwidth, storage throughput, shared caches, and kernel resources behave differently, and a design that caps CPU and memory can still suffer a noisy neighbor on I/O. Each control covers only the resource it governs. Evidence that the limits work therefore comes from a deliberately greedy workload in a controlled test environment: the limits hold, and other workloads’ latency stays within objectives.

The OCI runtime specification carries these settings in the container’s runtime configuration: the Linux section includes the namespaces to create or join, the cgroup path, resources such as memory limit, CPU shares, quota, period and PID limit, masked and read-only paths, and a seccomp profile. A higher-level tool translates its flags into that configuration, so for any running container the effective settings are in that configuration and in the cgroup files, not in the image.

Security boundaries

A shared kernel and the privileges around it

Because containers share the host kernel, a vulnerability in a kernel interface reachable from a container is a vulnerability for every container on the host, and a kernel crash takes down all of them. Namespaces and cgroups do not remove that exposure. What reduces it is limiting what a process may ask the kernel to do.

• Capabilities. Linux splits root’s power into capabilities. The manual page cautions against CAP_SYS_ADMIN in particular, because it covers an enormous range of operations. Docker starts containers with a restricted default set and an allowlist approach; --cap-drop removes capabilities, and --cap-add adds specific ones.
• System-call filtering. Docker’s default seccomp profile is an allowlist that, according to the documentation, disables around 44 of more than 300 system calls, and the documentation advises against changing it without need. You can supply a stricter profile, and seccomp=unconfined removes the filter.
• No new privileges. The no-new-privileges security option stops processes from gaining privileges, so setuid helpers such as su and sudo no longer elevate.
• Read-only root filesystem. --read-only makes the container’s root filesystem read-only except for specified volumes, which limits what a compromised process can persist.
• Privileged mode. --privileged gives the container all capabilities, access to all host devices, and relaxed AppArmor or SELinux confinement, which the documentation describes as nearly the same access to the host as processes outside containers. Use specific capabilities or devices instead.
• Non-root user. Run as a numeric non-root user, and combine it with a user namespace mapping where your platform supports one.

The least privilege principle applies to each item: start with all of them restricted, and add back only what the application is shown to need by testing. A practical way to find the needed set is to run the workload in a test environment with an intentionally small allowance, observe the failures, and extend deliberately, recording the reason for each exception.

The control plane of the container engine

The daemon that creates containers is a privileged component. The Docker security documentation says the daemon needs root unless rootless mode is used, that only trusted users should control it, and that it can map host directories into containers without restriction. Giving a process access to the daemon’s socket therefore amounts to giving it root-equivalent control of the host. This is why mounting the container engine’s socket into a workload container, a common convenience for build and tooling containers, deserves the same review as granting host administration. If a build or CI workload needs to build images, prefer a builder that does not require handing it the daemon.

A public case: a runtime escape

Container boundaries are enforced by software that can have defects. CVE-2019-5736, published in February 2019, concerned runc, the low-level runtime used by Docker and other products. The National Vulnerability Database describes it as allowing an attacker to overwrite the host’s runc binary, and so obtain root on the host, by exploiting file-descriptor mishandling related to /proc/self/exe when executing a command as root inside a new container with an attacker-controlled image or in an existing container that can be attached with docker exec. The record lists a CVSS v3.1 base score of 8.6 (high). These are the facts of the record. The interpretation for platform engineers is that running attacker-influenced images, or running workloads as root without a user namespace mapping, relied on the runtime having no defect at the boundary, and that the control available to operators was timely updating of the runtime along with limiting who can supply images. Mitigation of such cases combines a patched runtime, user namespace mapping, and not granting execution rights in containers to untrusted parties.

Choosing a stronger boundary

Whether ordinary containers are sufficient depends on who can run code in them. For a team’s own services after review, the combination above is a common baseline. For workloads that execute code written by untrusted parties, the shared kernel is a larger risk, and a design that interposes a virtual machine or a user-space kernel between the workload and the host is a different boundary with different costs. The decision depends on the threat model, performance needs, and operating capacity, and should be made deliberately and documented, in the spirit of the sandbox entry, which stresses that the name of a boundary says little about its actual restrictions and that it should be tested.

Diagnosing a container at run time

Diagnosis improves when you start from the part of the model that would produce the symptom, and check evidence from that layer before changing anything. The table maps common symptoms to the layer to inspect first. It is a starting point for investigation, not a decision tree that proves a cause.

Symptom	First layer to inspect	Evidence to collect
Container exits immediately	Image defaults and the process	Exit code (125, 126, 127 mean the command could not run), configured command and entrypoint, user, working directory, missing files from the image
Exit code 137	Memory cgroup, or an external kill	memory.events oom_kill counter, the engine’s recorded state for the container, kernel log, whether a stop request timed out
Stops slowly, then 137	PID 1 and signal handling	Whether the main process handles the stop signal, whether it waits on children, the grace period
Latency spikes with low average CPU	CPU quota	cpu.stat throttling counters against period count, request latency percentiles
Fork or thread creation errors	pids controller	pids.max compared with pids.current, application thread settings
Permission denied on a file	User, mounts, capabilities	Effective user, ownership and mode of the path, whether a mount shadows or is read-only, user namespace mapping
Files missing after adding a volume	Mount namespace	Mount list inside the container, what directory the volume covers
Cannot reach another service	Network namespace	Interfaces and routes inside the namespace, which network the container is attached to, published ports, name resolution
Slow neighbor effects on a host	Cgroup limits and shared resources	Per-container usage, limits on each resource, shared I/O and cache contention

Three diagnostic habits make this reliable. First, look at the container from inside and from outside. From inside, you see the namespaced view: the process list, mounts, interfaces, and the effective user. From outside, on the host, you see the cgroup files, the kernel log, and the engine’s recorded state. Disagreement between the two is information. Second, compare effective settings with intended settings. The image configuration says what was intended as a default, and the runtime configuration and the live cgroup files say what is in force. Third, prefer reading counters to guessing: the memory.events and cpu.stat counters record that a limit was hit, which a restart count cannot.

A short runbook for the most frequent case, an unexplained restart loop, can be written entirely from the above (not executed):

1. Record the exit code and the last log lines, and whether the container was stopped by the platform or died by itself.
2. If the code indicates a signal, read the memory cgroup’s memory.events and the kernel log for out-of-memory records, and compare peak usage with the limit.
3. If the process exits by itself, run the image with its configured command and user in a test environment and compare filesystem, mounts, and environment with what the platform supplies.
4. If it hangs on stop, check that PID 1 receives and handles the stop signal and consider an init wrapper that forwards signals and reaps children.
5. Change one setting at a time and record the result, so the fix is understood and not accidental.

Be careful about what an inspection tool can disturb. Entering a container’s namespaces with nsenter, attaching a debugging shell, or lifting limits to reproduce a problem can alter the state you are investigating, and each such action should be allowed only for authorized operators and recorded.

Limits of this model and what to verify

Several statements in this post depend on conditions worth restating. The description of namespaces and cgroups applies to Linux containers sharing a kernel; runtimes that use virtual machines or user-space kernels differ. Cgroup file names and semantics given here are for cgroup v2, and v1 systems have different interfaces. The default limits, profiles, and capability sets are those documented for Docker at the time of reading and can differ across container engines and versions, so confirm them for the engine you operate. The layer example is a hand-built toy that follows the documented structure and does not exercise a registry or a real builder. The CPU calculations simplify kernel scheduling. And none of the Linux commands was run for this post.

The settings in this post can be collected into an acceptance check on a service’s container configuration. Such a check asserts that the image is referenced by digest, that the effective user is non-root, that the root filesystem is read-only where possible, that capabilities beyond the needed set are dropped, that memory, CPU, and PID limits are set and measured, that no sensitive host path or engine socket is mounted, and that only the intended ports are published. It is useful to record which assertions are checked from the image and which from the live container, because the two can disagree. The next post, Kubernetes Controllers and Workload Lifecycle, shows how a platform takes these settings as declared intent and keeps a set of such containers running and replaced as conditions change.

References

• Linux man-pages, namespaces(7)
• Linux man-pages, pid_namespaces(7)
• Linux man-pages, mount_namespaces(7)
• Linux man-pages, cgroups(7)
• Linux man-pages, capabilities(7)
• Linux man-pages, unshare(1)
• Linux man-pages, nsenter(1)
• Linux kernel documentation, Control Group v2
• Open Container Initiative, Image specification: Manifest
• Open Container Initiative, Image specification: Layer
• Open Container Initiative, Image specification: Configuration
• Open Container Initiative, Runtime specification: Linux container configuration
• Docker documentation, Resource constraints
• Docker documentation, Docker Engine security
• Docker documentation, Seccomp security profiles
• Docker documentation, OverlayFS storage driver
• Docker documentation, Isolate containers with a user namespace
• Docker documentation, Running containers (exit status and runtime privilege)
• Docker documentation, docker container run reference
• Docker documentation, Bind mounts
• Docker documentation, Build cache invalidation
• Docker documentation, Build secrets
• Docker documentation, Bridge network driver
• NVD, CVE-2019-5736