Linux Processes, Resources, and Failure Diagnosis

A service that is supposed to answer 99 percent of requests within 200 milliseconds misses that target. The dashboard shows the miss and nothing about the cause. The cause could be a process that was killed and restarted, a process that is alive but starved of CPU, a memory limit that is too low, a disk that is slow, a log line that never reached anyone because it sat in a buffer, or a file that the process is not allowed to open. Each of these has a different fix, and each leaves a different trace. Reading the trace correctly is a skill built on a small number of operating-system ideas.

This article covers those ideas as they apply to Linux-based platforms: processes, signals and permissions; CPU, memory and I/O; logs and resource limits; and a method for diagnosing and reproducing failures. The previous article, Platform Engineering: From Consumer Needs to Operated Services, defined a capability by a contract with a measurable guarantee and assigned an owner who answers when the guarantee is missed. This article assumes you took from it that a promise is stated as a service level indicator (the fraction of good events among valid events) and that someone owns what happens next. The operating system layer here is where that owner goes to find out why. The core explanation for each topic is in this article and does not depend on the earlier one.

A note on how the examples were produced, because it affects what they prove. The programs below were run on macOS (Darwin 25.6, Python 3.12) as an unprivileged user, using only the Python standard library and the shell. Signals, exit statuses, permission checks, fork and wait, resource limits and buffering are POSIX behaviors that Linux shares, so those results transfer. Linux-specific mechanisms (/proc, control groups, the kernel’s out-of-memory killer, pressure stall information) cannot be observed on macOS and were not executed. They are described from the Linux manual pages and kernel documentation cited in the references, and one small Linux-oriented parser is exercised only on made-up sample text, which is stated where it appears. Some values on macOS differ from Linux, and those differences are pointed out where they matter. All code was written for this article. The cgread.py listing is the only one that needs Linux to read real files, and its self-test runs anywhere on its invented sample text.

Processes: what the kernel tracks and how a process ends

A process is a running instance of a program, together with the kernel’s bookkeeping about it: an identifier (PID), a parent, an address space, a table of open file descriptors, credentials (user and group identity), signal dispositions, and resource limits. The shell you type commands into is a process. When it runs a command, it creates a child process with fork and then replaces the child’s program with the requested one using exec. Understanding which attributes survive those two steps explains many surprises. The Linux manual for signal(7) states that a child created by fork inherits copies of its parent’s signal dispositions, and that execve resets handlers for caught signals to their default while leaving ignored signals ignored. The manual for getrlimit(2) states that resource limits are inherited through fork and preserved across execve, which is how a shell can constrain the commands it launches. An environment variable is another inherited attribute, which is why a job started by a scheduler can behave differently from the same command typed in a shell with a different environment.

Process states and what they tell you

On Linux, /proc/<pid>/status is a text file describing a process. The proc_pid_status(5) manual lists its fields, including a State field with values such as R (running), S (sleeping), D (disk sleep), T (stopped) and Z (zombie), the memory fields VmSize, VmRSS, RssAnon and RssFile, the thread count, and counts of voluntary and involuntary context switches. For diagnosis, state is the first coarse filter. A process that is S (sleeping) is waiting for something, and the question becomes what. A process in R is runnable or running and is a candidate for a CPU problem. A process in D is in disk sleep, which is typically an uninterruptible wait on I/O; a process stuck in that state may not respond to signals until the wait ends. A process in Z has already finished and is waiting for its parent.

Zombies and orphans

When a process exits, the kernel keeps a small record (its PID, termination status and resource usage) until the parent collects it with the wait family of calls. The wait(2) manual calls a child that has terminated but has not been waited for a zombie. If a parent never waits, zombies accumulate in the process table. The manual also states that when a parent terminates, its children are adopted by init(1) or the nearest subreaper process, which then waits for them. The following program forks a child that exits with status 7, deliberately delays the parent’s wait, and asks ps about the child before and after.

import os, subprocess, time
pid = os.fork()
if pid == 0:
    os._exit(7)                      # child exits immediately with status 7
time.sleep(0.3)                      # parent has not called wait yet
print(subprocess.run(["ps", "-o", "pid,ppid,stat,comm", "-p", str(pid)], capture_output=True, text=True).stdout.strip())
_, status = os.waitpid(pid, 0)
print("waitpid:", "exited" if os.WIFEXITED(status) else "signaled", "code", os.WEXITSTATUS(status))
r = subprocess.run(["ps", "-o", "pid,stat,comm", "-p", str(pid)], capture_output=True, text=True)
print("after wait, ps found the pid:", r.returncode == 0)

Output on macOS (the ps column layout is the macOS one; the Z state and the <defunct> marker are the same idea on Linux):

PID  PPID STAT COMM
29174 29160 Z    <defunct>
waitpid: exited code 7
after wait, ps found the pid: False

The child’s state is Z and its command shows as <defunct> until the parent calls waitpid. After the call the PID no longer exists. The practical consequence for a platform is that a long-running supervisor that starts children and does not collect their status leaks process-table entries, and the leak is invisible in CPU and memory graphs. A zombie uses almost no resources by itself; the risk is the table filling up, as the manual notes, and the loss of information about how the child ended.

Exit status and signals are different ways to end

A process can end because it called exit with a number, or because a signal terminated it. The wait macros distinguish them: WIFEXITED and WEXITSTATUS for a normal exit (the low 8 bits of the status), and WIFSIGNALED and WTERMSIG for termination by signal. Python’s subprocess reports the second case as a negative return code, so -9 means signal 9. A shell cannot return a negative number, so Bash uses the convention that a command ended by fatal signal number N has exit status 128+N, that a command that is not found returns 127, and that a command that is found but not executable returns 126 (all from the Bash manual). The exit status glossary entry covers how to read and save the value in scripts. The following shell run shows each case.

sh: line 1: 29039 Terminated: 15          sleep 30
TERM status=143
sh: line 1: 29040 Killed: 9               sleep 30
KILL status=137
sh: /nonexistent/cmd: No such file or directory
missing status=127
sh: /tmp/x_noexec_29037.sh: Permission denied
noexec status=126

Status 143 is 128+15 (SIGTERM) and 137 is 128+9 (SIGKILL). The 126 case here arises because the script file was created without the execute bit, which also shows how a permission problem appears as an exit status. Seeing 137 in a platform’s status field therefore means the process was ended by SIGKILL, but it does not say who sent it. Candidates include a supervisor escalating after a graceful stop timed out, a person, or the kernel’s out-of-memory killer. Kubernetes documents that a container that exceeds its memory limit may be killed by the kernel and reported with exit code 137 and the reason OOMKilled. Treat 137 as a prompt to find the sender, not as a diagnosis.

Signals: asking a process to stop, and what it may do

A signal is a small asynchronous notification delivered to a process. Each signal has a default action, and a process can change how it handles most of them. Per the signal(7) manual, the default action of SIGTERM, SIGINT, SIGHUP, SIGPIPE and SIGKILL is to terminate the process, SIGSEGV terminates it and produces a core dump, SIGCHLD is ignored by default, and SIGSTOP stops the process. On x86 the numbers are SIGHUP 1, SIGINT 2, SIGKILL 9, SIGSEGV 11 and SIGTERM 15. The decisive property is that SIGKILL and SIGSTOP cannot be caught, blocked or ignored. Every other termination request can be handled by the application, and that choice is what separates a graceful stop from an abrupt one.

SIGTERM is the conventional polite request to stop. A well-behaved service catches it, stops accepting new work, finishes or hands off what is in flight, and exits with status 0. SIGKILL is the enforcement: the process gets no chance to run any code. A supervisor that wants both safety and a bound on waiting sends SIGTERM, waits for a grace period, and sends SIGKILL if the process is still alive. The graceful shutdown glossary entry describes the same sequence for Kubernetes, including the point that hooks and cleanup share one grace budget. The program below demonstrates three child behaviors under a SIGTERM: default handling, a handler that cleans up, and ignoring the signal.

import signal, sys, time, os
mode = sys.argv[1]
def on_term(signum, frame):
    print("child: got SIGTERM, cleaning up", flush=True)
    sys.exit(0)
if mode == "graceful":
    signal.signal(signal.SIGTERM, on_term)
elif mode == "ignore":
    signal.signal(signal.SIGTERM, signal.SIG_IGN)
print("child: ready pid", os.getpid(), flush=True)
while True: time.sleep(0.1)

# ---- sig_demo.py (the parent) ----
import subprocess, signal, time, sys
def run(mode, sig):
    p = subprocess.Popen([sys.executable, "sig_child.py", mode], stdout=subprocess.PIPE, text=True)
    p.stdout.readline()                      # wait until the child is ready
    p.send_signal(sig)
    try:
        out, _ = p.communicate(timeout=1.0)
        how = "exited by itself"
    except subprocess.TimeoutExpired:
        p.kill(); out, _ = p.communicate(); how = "still alive after 1s, sent SIGKILL"
    print(f"{mode:9s} + {signal.Signals(sig).name:8s} -> returncode={p.returncode:4d} ({how}) output={out.strip()!r}")
run("default", signal.SIGTERM)
run("graceful", signal.SIGTERM)
run("ignore", signal.SIGTERM)
try:
    signal.signal(signal.SIGKILL, lambda *a: None)
except OSError as e:
    print("catching SIGKILL:", type(e).__name__, e)

Output:

default   + SIGTERM  -> returncode= -15 (exited by itself) output=''
graceful  + SIGTERM  -> returncode=   0 (exited by itself) output='child: got SIGTERM, cleaning up'
ignore    + SIGTERM  -> returncode=  -9 (still alive after 1s, sent SIGKILL) output=''
catching SIGKILL: OSError [Errno 22] Invalid argument

The default child died from the signal (-15) without running any application code. The graceful child ran its handler, printed its message and returned 0. The child that ignored SIGTERM was still alive after one second, so the parent sent SIGKILL and the return code became -9. The last line shows that attempting to install a handler for SIGKILL fails; on this macOS system the error is EINVAL, consistent with the manual’s statement that SIGKILL cannot be caught.

What can go wrong with this protocol in a platform is worth listing, because each item recurs.

• The signal reaches the wrong process. If an application is started through a wrapper such as a shell script, the signal may be delivered to the wrapper rather than the application, depending on how the wrapper starts it. The graceful shutdown entry makes the same point for containers: check that the main process actually receives the stop signal.
• The handler never finishes. A handler that waits for work that depends on a stopped resource turns a graceful stop into a timeout followed by SIGKILL. The supervisor’s grace period is the bound, and a stop that always hits the bound should be investigated.
• Work is lost on SIGKILL. Anything held only in process memory is gone, including buffered log output (shown below), partially written files and in-memory queues. Design for it: write data in a form that can be retried, and make operations safe to repeat.
• Dispositions are inherited unexpectedly. Because ignored signals stay ignored across exec, a child started by a parent that ignored a signal ignores it too, unless the child resets it.

Permissions: the checks that decide whether an operation is allowed

Every file operation by a process is checked against the process’s credentials and the file’s metadata. The path_resolution(7) manual describes the rules. Each file has an owner, a group and nine permission bits: read, write and execute for the owner, for the group, and for everyone else. The kernel checks one class only. If the process’s user matches the owner it uses the owner bits, otherwise if the group matches (the process’s group or supplementary groups) it uses the group bits, and otherwise it uses the other bits. This is a first-match rule, so an owner who has no permission is denied even when the group and other bits would allow access. On Linux the user identity used for file checks is the filesystem user ID, which is normally equal to the effective user ID.

Directories add a rule that surprises people. To reach a file by path, the process needs search permission on every directory in the path, and for directories search permission is the execute bit. If that is missing, the lookup fails with EACCES before the file’s own bits are consulted. Read permission on a directory is a separate thing: it governs listing the names inside. The following program probes these rules with a file and a directory, running as an unprivileged user.

import os, stat, tempfile, errno
d = tempfile.mkdtemp(); sub = os.path.join(d, "data"); os.mkdir(sub)
f = os.path.join(sub, "report.txt"); open(f, "w").write("x")
def try_open(label, mode="r"):
    try:
        open(f, mode).close(); print(f"{label:55s} ok")
    except OSError as e:
        print(f"{label:55s} {errno.errorcode[e.errno]} ({e.strerror})")
os.chmod(f, 0o644); try_open("file 0644, read")
os.chmod(f, 0o444); try_open("file 0444, write", "w")
os.chmod(f, 0o000); try_open("file 0000, read")
os.chmod(f, 0o644)
os.chmod(sub, 0o644); try_open("file 0644 but parent dir 0644 (no x), read")
os.chmod(sub, 0o111); print(f"{'dir 0111 (x only): listing':55s}", end=" ")
try: os.listdir(sub); print("ok")
except OSError as e: print(errno.errorcode[e.errno])
try_open("dir 0111, open known path, read")
os.chmod(sub, 0o755); os.remove(f); os.rmdir(sub); os.rmdir(d)

file 0644, read                                         ok
file 0444, write                                        EACCES (Permission denied)
file 0000, read                                         EACCES (Permission denied)
file 0644 but parent dir 0644 (no x), read              EACCES (Permission denied)
dir 0111 (x only): listing                              EACCES
dir 0111, open known path, read                         ok

Reading the rows: a file with mode 0444 cannot be opened for writing and a file with mode 0000 cannot be read, as expected. A file with permissive mode 0644 in a directory lacking the execute bit is still unreachable, which is the directory rule. A directory with only the execute bit refuses listing but allows opening a file whose name is already known. That last row is the practical distinction between being able to enumerate and being able to reach. These results hold for an unprivileged user; privileged processes follow additional rules that this article does not cover.

How this appears on a platform follows from the rules. A containerized service that runs as a different user than the one that owns a mounted volume sees EACCES even though the files look readable to the person who inspects them. This is a common diagnosis pattern, stated here as an inference from the rules above and not as a measured frequency. The investigation is mechanical: find the process’s user and groups, find the owner, group and mode of the file and of every directory on the path, and apply the first-match rule. A “file not found” error (ENOENT) is a different failure, and confusing the two sends people to look at the wrong thing.

CPU, memory and I/O: three resources that fail differently

A process uses CPU time, memory and I/O bandwidth, and each of these runs out in a different way. CPU contention produces slowness. Memory exhaustion typically produces death. I/O contention produces waiting that looks like slowness but consumes no CPU. Distinguishing them early prevents tuning the wrong thing.

CPU: busy time versus elapsed time

The first discriminating measurement is CPU time compared with elapsed (wall-clock) time. If a process is slow and its CPU time is close to its elapsed time, it is computing the entire time, and speed depends on how much work it does or how much CPU it gets. If its CPU time is much smaller than its elapsed time, it spends time waiting for something else. The program below measures a busy loop, a sleep and a file write with getrusage, which reports user time (in application code) and system time (in the kernel on the process’s behalf).

import os, time, resource, subprocess, tempfile, sys, multiprocessing

def report(label, fn):
    t0 = time.perf_counter(); r0 = resource.getrusage(resource.RUSAGE_SELF)
    fn()
    wall = time.perf_counter() - t0; r1 = resource.getrusage(resource.RUSAGE_SELF)
    user, sys_ = r1.ru_utime - r0.ru_utime, r1.ru_stime - r0.ru_stime
    print(f"{label:28s} wall={wall:5.2f}s user={user:5.2f}s sys={sys_:5.2f}s cpu/wall={(user+sys_)/wall:4.2f}")

def burn():
    end = time.perf_counter() + 1.0
    while time.perf_counter() < end: pass
def sleepy(): time.sleep(1.0)
def write_fsync():
    with tempfile.NamedTemporaryFile(dir=".") as f:
        chunk = os.urandom(1024 * 1024)
        for _ in range(256): f.write(chunk)
        f.flush(); os.fsync(f.fileno())

def rss_kb():
    return int(subprocess.run(["ps", "-o", "rss=", "-p", str(os.getpid())], capture_output=True, text=True).stdout)

if __name__ == "__main__":
    report("busy loop 1s", burn)
    report("sleep 1s", sleepy)
    report("write 256 MiB + fsync", write_fsync)
    base = rss_kb()
    buf = bytearray(512 * 1024 * 1024)            # ask for 512 MiB; zero-filled, pages not yet written
    after_alloc = rss_kb()
    for i in range(0, len(buf), 4096): buf[i] = 1  # write one byte per 4 KiB page
    after_touch = rss_kb()
    print(f"RSS baseline={base/1024:7.1f} MiB | after bytearray(512MiB)={after_alloc/1024:7.1f} MiB | after touching pages={after_touch/1024:7.1f} MiB")
    print("ru_maxrss raw value:", resource.getrusage(resource.RUSAGE_SELF).ru_maxrss, "(bytes on macOS, kilobytes on Linux)")

busy loop 1s                 wall= 1.00s user= 0.95s sys= 0.02s cpu/wall=0.97
sleep 1s                     wall= 1.00s user= 0.00s sys= 0.00s cpu/wall=0.00
write 256 MiB + fsync        wall= 0.15s user= 0.00s sys= 0.10s cpu/wall=0.68
RSS baseline=   11.8 MiB | after bytearray(512MiB)=  523.7 MiB | after touching pages=  523.7 MiB
ru_maxrss raw value: 549224448 (bytes on macOS, kilobytes on Linux)

The first two lines are the point. The busy loop used 0.97 of every elapsed second as CPU, and the sleeping process used 0.00. A latency problem looks the same from outside for both, which is why the ratio is the first thing to compute. The write line shows sys time (0.10 s) because writes and fsync are kernel work, and a CPU-to-wall ratio of 0.68. On this machine the write finished in 0.15 seconds, probably served by a fast local disk and caches, so this run does not show an I/O bottleneck. It shows the measurement, not a typical disk speed, and the number should not be generalized.

On a shared platform there is a second way to be slow with spare capacity on the host: the CPU limit. The cgroup v2 kernel documentation describes cpu.max as two numbers, a quota and a period, meaning the group may use up to the quota in each period; the default is max 100000, which is unlimited within a 100 ms period. A group that uses its quota early in a period is paused until the next one. The group’s cpu.stat file records nr_periods, nr_throttled and throttled_usec, so the fraction of enforcement periods in which throttling occurred is nr_throttled / nr_periods over the interval of interest. Kubernetes documents that a container approaching its CPU limit is throttled and not terminated. The small parser below reads such files. It was run only on invented sample text, since it needs Linux to read real files.

"""Read Linux cgroup v2 and /proc files for one process. Linux only: the parsing is run here on made-up sample text."""
import pathlib

def parse_flat_keyed(text):
    """Files such as cpu.stat and memory.events hold one 'key value' pair per line."""
    return {k: int(v) for k, v in (line.split() for line in text.strip().splitlines())}

def parse_status(text):
    out = {}
    for line in text.splitlines():
        k, _, v = line.partition(":")
        out[k] = v.strip()
    return out

def cgroup_dir(pid):
    # cgroup v2 entry in /proc/<pid>/cgroup looks like "0::/some/path"
    for line in pathlib.Path(f"/proc/{pid}/cgroup").read_text().splitlines():
        hier, _, path = line.split(":", 2)
        if hier == "0": return pathlib.Path("/sys/fs/cgroup") / path.lstrip("/")
    raise RuntimeError("not a cgroup v2 process")

def throttle_ratio(cpu_stat_before, cpu_stat_after):
    d = {k: cpu_stat_after[k] - cpu_stat_before[k] for k in ("nr_periods", "nr_throttled", "throttled_usec")}
    return d, (d["nr_throttled"] / d["nr_periods"] if d["nr_periods"] else 0.0)

if __name__ == "__main__":
    # Synthetic sample, NOT measured from any system. Two reads of cpu.stat taken 10 s apart (100 ms period => 100 periods).
    a = parse_flat_keyed("usage_usec 5000000\nuser_usec 4800000\nsystem_usec 200000\nnr_periods 1000\nnr_throttled 100\nthrottled_usec 2000000")
    b = parse_flat_keyed("usage_usec 14000000\nuser_usec 13600000\nsystem_usec 400000\nnr_periods 1100\nnr_throttled 160\nthrottled_usec 5600000")
    delta, ratio = throttle_ratio(a, b)
    print(delta, f"throttled in {ratio:.0%} of periods")
    st = parse_status("Name:\tworker\nState:\tS (sleeping)\nVmSize:\t  812340 kB\nVmRSS:\t  204800 kB\nThreads:\t12")
    print(st["State"], st["VmRSS"], st["Threads"])

{'nr_periods': 100, 'nr_throttled': 60, 'throttled_usec': 3600000} throttled in 60% of periods
S (sleeping) 204800 kB 12

The numbers in the sample are made up to exercise the arithmetic: between two reads, 100 periods elapsed and 60 of them had throttling, so the group was throttled in 60 percent of periods. A throttled fraction that high while average CPU use looks modest is the signature of a limit that is too low for the workload’s bursts. This is the case where a service looks slow, the host has idle CPU, and raising the limit helps. It cannot be established from average utilization alone. The resource requests and limits glossary entry explains the distinction between a request used for scheduling and a limit enforced at run time.

Memory: virtual size, resident size, and limits

Memory has two sizes that are easy to confuse. Virtual size (VmSize) counts the address space a process has mapped. Resident size (VmRSS) counts the pages currently held in physical memory, which the manual defines as the sum of RssAnon (anonymous memory), RssFile (file mappings) and shared memory. A process can map far more than it uses. The next program maps 512 MiB anonymously and reports resident size at each step, using ps.

import mmap, os, subprocess
def rss(): return int(subprocess.run(["ps","-o","rss=","-p",str(os.getpid())],capture_output=True,text=True).stdout)/1024
size = 512*1024*1024
print(f"baseline RSS            {rss():7.1f} MiB")
m = mmap.mmap(-1, size)              # anonymous mapping: address space reserved
print(f"after mmap 512 MiB      {rss():7.1f} MiB   (mapped size {size/2**20:.0f} MiB)")
for i in range(0, size//2, 4096): m[i] = 1   # touch first half, one byte per page
print(f"after touching half     {rss():7.1f} MiB")
for i in range(size//2, size, 4096): m[i] = 1
print(f"after touching all      {rss():7.1f} MiB")

baseline RSS                9.2 MiB
after mmap 512 MiB          9.1 MiB   (mapped size 512 MiB)
after touching half       265.2 MiB
after touching all        521.2 MiB

Mapping 512 MiB changed resident size by a negligible amount. Touching half the pages brought it to about 265 MiB and touching all pages to about 521 MiB. Memory is committed lazily, when pages are first written, so a process’s resident size grows during execution and a limit is hit during execution, not at startup. This behavior was observed on macOS only. The Linux fields VmSize and VmRSS distinguish the same two sizes, but this run does not show how Linux accounts for them. One further small trap appears in the earlier output: ru_maxrss printed 549224448, which is bytes on macOS and kilobytes on Linux. A monitoring script that reads that field must handle the unit.

What happens at a limit is the part that matters for operations. The cgroup v2 documentation describes memory.max as the hard limit: if usage reaches it and cannot be reduced, the OOM killer is invoked within that cgroup, after the kernel first tries to reclaim memory. memory.high is a throttle limit: above it, processes in the group are throttled and put under heavy reclaim pressure, without invoking the OOM killer. The memory.events file counts how often each of these happened, including oom and oom_kill. A process can influence its exposure: per proc_pid_oom_score_adj(5), the value is added to the kernel’s badness score when choosing a victim, ranges from -1000 to +1000, and -1000 disables OOM killing for the process. Kubernetes documents that exceeding a memory limit may result in the container being killed with exit code 137 and reason OOMKilled, and that memory limits are enforced reactively, so a container may briefly exceed one.

A memory problem therefore shows up as a restart, not slowness, which is the opposite of CPU. The evidence is the exit status or termination reason, the oom_kill counter in memory.events and kernel log messages. A slow rise in resident size before each restart suggests a leak or a growing cache, and a sudden jump suggests a single large allocation. The limit that was hit may be the container’s, not the machine’s, so the host can have free memory while the service dies. The glossary entries on resource exhaustion and noisy neighbors cover the shared-platform side, where one workload’s consumption degrades others.

I/O: waiting that costs no CPU

A process waiting for disk or network is not using CPU, so CPU graphs look calm while latency rises. In the State field this appears as S or D rather than R. The kernel exposes pressure stall information under /proc/pressure/cpu, /proc/pressure/memory and /proc/pressure/io, and, with control groups, in cpu.pressure, memory.pressure and io.pressure inside each group. Each file has a some line and a full line with averages over 10, 60 and 300 seconds and a cumulative total in microseconds. The documentation defines some as the share of time in which at least some tasks were stalled on the resource, and full as the share of time in which all non-idle tasks were stalled at once. A rising some value for I/O indicates contention that a CPU metric would miss, and a nonzero full for memory indicates thrashing. The cgroup io.max file can limit read and write bytes and operations per second for a device, so I/O can be throttled by policy just as CPU can.

File descriptors are a related I/O resource. Every open file, socket and pipe consumes a descriptor, and the per-process limit RLIMIT_NOFILE causes EMFILE when exceeded. Services that open connections per request without closing them can reach this limit long before they exhaust memory. The diagnosis tool in the last section reproduces it by lowering the limit.

Symptom	Resource	Evidence to collect first	What the limit does
Slow, CPU time close to wall time	CPU	CPU-to-wall ratio, cpu.stat throttling counters	Throttles (cgroup quota); with RLIMIT_CPU sends SIGXCPU, then SIGKILL
Restarts, status 137, reason OOMKilled	Memory	Termination reason, memory.events oom_kill, resident size trend	Reclaim, then the OOM killer in the group; memory.high throttles instead
Slow, low CPU, state S or D	I/O or a dependency	Pressure stall io lines, what the process is waiting on	io.max throttles bandwidth or operations
EMFILE or Too many open files	File descriptors	Count of open descriptors against RLIMIT_NOFILE	Further opens fail with an error
fork fails with EAGAIN	Processes	Process count per user or pids.max	Further process creation fails

Logs and resource limits

Logs: what reaches a reader, and when

A process’s diagnostic output normally goes to standard output and standard error, the conventional descriptors 1 and 2. The standard streams and redirection entry explains how a platform can connect them to a terminal, a file or a pipe. Platforms commonly collect from those streams or from log files. Two facts determine whether the evidence exists when you need it. First, a process may hold output in a buffer that is lost if it is killed. Second, even delivered output may be in a form that is hard to query.

The first fact can be tested directly. The program below starts a child that prints three lines to a pipe and then sleeps, kills it with SIGKILL after 0.7 seconds, and shows what the parent captured under three configurations.

import subprocess, sys, time

def run(args, label):
    p = subprocess.Popen(args, stdout=subprocess.PIPE, text=True)
    time.sleep(0.7); p.kill(); out, _ = p.communicate()
    print(f"{label:34s} captured after SIGKILL: {out!r}")

child = "import time\nfor i in range(3): print('line', i)\ntime.sleep(5)"
run([sys.executable, "-c", child], "stdout is a pipe, default buffering")
run([sys.executable, "-u", "-c", child], "python -u (unbuffered)")
child2 = "import time\nfor i in range(3): print('line', i, flush=True)\ntime.sleep(5)"
run([sys.executable, "-c", child2], "print(..., flush=True)")

stdout is a pipe, default buffering captured after SIGKILL: ''
python -u (unbuffered)             captured after SIGKILL: 'line 0\nline 1\nline 2\n'
print(..., flush=True)             captured after SIGKILL: 'line 0\nline 1\nline 2\n'

With default buffering and a pipe, nothing was captured: the three lines were in the child’s buffer when SIGKILL arrived, and SIGKILL gives the process no chance to flush. Running with python -u or calling print with flush=True delivered all three lines. The Python documentation states that -u forces stdout and stderr to be unbuffered, and that since version 3.7 this applies to the text layer. The general lesson is not specific to Python. When a process writes to a pipe or file instead of a terminal, many runtimes buffer by default, and the last lines before an abrupt kill are the lines most likely to explain it. A service should flush important events, especially errors, or use a logging library whose handler writes each event as it occurs.

The second fact is about structure. Structured logging records an event as named fields, such as a request identifier, a status and a duration, so a collector can filter by field without parsing prose. Stable field names allow queries to survive changes in wording, and an identifier carried across log lines lets you reconstruct what happened to one request. A structured record still needs care about secrets and personal data, because JSON encoding does not remove them. When logs are written to files instead of streams, rotation matters: a collector that follows a path can lose or repeat lines when the file is renamed or truncated, and a restart checkpoint must identify the file generation as well as a position. For a platform, the contract with consumers should say where logs go, in which form, how long they are kept and what is guaranteed about completeness.

Resource limits: where they are set and what they do on breach

Limits exist at several layers, and a failure may come from any of them. Distinguishing the layer is part of diagnosis.

• Per-process limits (rlimits). Set with setrlimit and described in getrlimit(2). Each has a soft limit, which the kernel enforces, and a hard limit, which is the ceiling for the soft limit; an unprivileged process can raise its soft limit up to the hard limit and can lower the hard limit irreversibly. Limits are inherited by children and preserved across exec. Documented breach behaviors include EMFILE for RLIMIT_NOFILE, SIGXCPU at the soft limit of RLIMIT_CPU (repeating each second until the hard limit, where SIGKILL ends the process), ENOMEM or a stack-growth failure for RLIMIT_AS, and EAGAIN from fork for RLIMIT_NPROC, which is enforced per real user ID, not per process.
• Control group limits. Described in the cgroups(7) manual and the cgroup v2 kernel documentation: controllers for CPU, memory, I/O and process count apply to a group of processes. A process is placed in a group by writing its PID to cgroup.procs, and in v2 a single unified hierarchy holds all controllers. These are the limits that container platforms use. They differ from rlimits in being shared by every process in the group, and in having different breach behavior (throttle, reclaim, OOM kill, EAGAIN on pids.max).
• Platform policy limits. Requests and limits in a scheduler’s configuration, quotas per team, and rate limits at an API boundary. A request is used for placement decisions and a limit constrains run-time use; neither is a prediction of peak use.

Behavior at these limits is why a consistent diagnosis needs the layer. The same symptom (a process that disappears) can be an application bug, an rlimit, a cgroup memory limit or an operator action, and each leaves a different trace. In the demonstration below, the CPU rlimit and the descriptor rlimit are applied to children and show up as a signal and an error respectively. Memory limits through RLIMIT_AS were not demonstrated on macOS, where enforcement could not be exercised here, so the Linux behavior is documented, not shown.

Diagnosing and reproducing failures

Diagnosis is a loop: observe, classify, form a hypothesis that predicts something checkable, check it, and repeat. Reproduction is the step that turns a hypothesis into evidence, since a failure that can be triggered on demand can be tested against a fix. The minimal reproducible example idea applies: reduce the failing scenario until removing anything more makes the failure disappear, and keep the reduced case as a regression test afterwards.

A classification order

1. How did the process end? Distinguish a normal exit, a nonzero exit status, and termination by a signal. This single fact eliminates whole categories of cause. A nonzero status with a message points at the application, and a signal points at the sender.
2. If a signal, which one and from whom? SIGSEGV and SIGABRT originate in the process. SIGTERM followed by SIGKILL suggests a stop that did not complete. SIGKILL alone needs a sender: a supervisor, an operator, or the kernel’s OOM handling, the last of which leaves counters and log messages.
3. If it did not end, what is it doing? Check state, then CPU time against wall time, then pressure stall information and what the process waits on.
4. Which limit applies? Identify limits at the process, group and platform layers and compare usage with each.
5. What does the last output say? Check the final log lines, remembering that buffered output may be missing if the process was killed.

A harness that classifies termination

The following program runs a command under explicit limits and reports how it ended. It is a small illustration of steps 1 and 2 and of reproduction: the limits are stated in the call, so a scenario is repeatable. It uses preexec_fn to set limits in the child. The Python documentation warns that preexec_fn is not safe in the presence of threads, which is acceptable in this single-threaded script and a reason to prefer other mechanisms in real tools.

"""Run a command under explicit limits and classify how it ended (illustrative, POSIX, stdlib only)."""
import resource, signal, subprocess, sys, time, json

HINTS = {
    signal.SIGSEGV: "invalid memory access in the process",
    signal.SIGABRT: "the process called abort(), often after a failed internal check",
    signal.SIGXCPU: "RLIMIT_CPU soft limit reached",
    signal.SIGKILL: "killed and not catchable: a supervisor, the OOM killer, or an earlier SIGTERM ignored",
    signal.SIGTERM: "asked to terminate and did not handle the signal",
}

def limiter(cpu, nofile):
    def apply():
        if cpu: resource.setrlimit(resource.RLIMIT_CPU, (cpu, cpu + 1))
        if nofile: resource.setrlimit(resource.RLIMIT_NOFILE, (nofile, resource.getrlimit(resource.RLIMIT_NOFILE)[1]))
    return apply

def run(name, code, cpu=None, nofile=None, wall=5.0):
    t0 = time.perf_counter()
    p = subprocess.Popen([sys.executable, "-c", code], stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                         text=True, preexec_fn=limiter(cpu, nofile))
    timed_out = False
    try:
        out, err = p.communicate(timeout=wall)
    except subprocess.TimeoutExpired:
        timed_out = True; p.terminate()
        try: out, err = p.communicate(timeout=1)
        except subprocess.TimeoutExpired: p.kill(); out, err = p.communicate()
    rc = p.returncode; el = time.perf_counter() - t0
    if timed_out: kind = f"wall-clock timeout ({wall}s), then signal {signal.Signals(-rc).name if rc < 0 else rc}"
    elif rc < 0:
        s = signal.Signals(-rc); kind = f"killed by {s.name}: {HINTS.get(s, 'see signal(7)')}"
    elif rc == 0: kind = "exited normally"
    else:
        tail = (err.strip().splitlines() or [""])[-1]
        kind = f"exited with status {rc}; last stderr line: {tail!r}"
    print(json.dumps({"case": name, "returncode": rc, "elapsed_s": round(el, 2), "diagnosis": kind}))

run("ok", "print('done')")
run("exit-3", "import sys; sys.stderr.write('config missing\\n'); sys.exit(3)")
run("abort", "import os; os.abort()")
run("segfault", "import ctypes; ctypes.string_at(0)")
run("hang", "import time; time.sleep(60)", wall=1.0)
run("cpu-limit", "while True: pass", cpu=1)
run("fd-limit", "fs=[open('/dev/null') for _ in range(100000)]", nofile=32)

The output of one run:

{"case": "ok", "returncode": 0, "elapsed_s": 0.04, "diagnosis": "exited normally"}
{"case": "exit-3", "returncode": 3, "elapsed_s": 0.04, "diagnosis": "exited with status 3; last stderr line: 'config missing'"}
{"case": "abort", "returncode": -6, "elapsed_s": 0.03, "diagnosis": "killed by SIGABRT: the process called abort(), often after a failed internal check"}
{"case": "segfault", "returncode": -11, "elapsed_s": 0.04, "diagnosis": "killed by SIGSEGV: invalid memory access in the process"}
{"case": "hang", "returncode": -15, "elapsed_s": 1.01, "diagnosis": "wall-clock timeout (1.0s), then signal SIGTERM"}
{"case": "cpu-limit", "returncode": -24, "elapsed_s": 1.03, "diagnosis": "killed by SIGXCPU: RLIMIT_CPU soft limit reached"}
{"case": "fd-limit", "returncode": 1, "elapsed_s": 0.05, "diagnosis": "exited with status 1; last stderr line: \"OSError: [Errno 24] Too many open files: '/dev/null'\""}

Each line is one failure shape. exit-3 ended by itself and left a message, so the application chose to stop. abort and segfault were terminated by SIGABRT and SIGSEGV, which originate in the process. hang never finished, so the harness’s wall-clock timeout sent SIGTERM; the child did not handle it, and its default action ended it. cpu-limit was ended by SIGXCPU after the one-second CPU limit, confirming the documented behavior of the soft limit. fd-limit exited with status 1 and its last stderr line named Errno 24, Too many open files, which is EMFILE. Only the cases that were run are claimed: an out-of-memory kill, a cgroup throttle and a D-state hang could not be reproduced on this system.

Notice what the harness cannot know. It cannot say why fd-limit opened so many files; it reports that the limit was reached and where. The hint table is a set of guesses attached to signal numbers, and a hint is a hypothesis to test, not a finding. For SIGKILL in particular, the table lists several candidates because the signal itself does not identify the sender.

An illustrative end-to-end diagnosis

The following scenario is hypothetical and constructed for this article. Suppose a service’s SLI is missing its target, and its platform reports that the process restarts at irregular intervals with exit status 137.

1. Status 137 is 128+9, so the process was terminated by SIGKILL. The sender is unknown. Candidates: a supervisor escalating after a failed stop, a person, or the kernel’s memory handling.
2. Check the platform’s termination reason. If it says OOMKilled, the memory limit is the leading hypothesis. If the reason is a failed health probe followed by a restart, the cause may instead be slowness, with SIGKILL as the supervisor’s last step.
3. If memory is the hypothesis, read the group’s memory.events. A rising oom_kill count confirms the kernel killed a process there. Compare resident size over time just before each restart to distinguish a gradual leak from a sudden spike.
4. Reproduce under a lower memory limit with the same workload and see whether the failure reappears with the same termination reason. If the restart does not reproduce, the hypothesis is not yet established.
5. After a fix (a smaller cache, a raised limit with a documented reason, or a bug fix), repeat the reproduction and watch the SLI over a window longer than the restart interval.

Every step produces something that could contradict the hypothesis. That is the property that separates diagnosis from a plausible story: a story survives any evidence, and a hypothesis states what would show it wrong.

Limits of what this article shows

• The runnable examples ran on macOS. They demonstrate POSIX behavior that Linux shares, but not Linux’s /proc, cgroups, OOM killer or pressure stall information, which are described from documentation and not observed here. Linux can differ in places, such as units in ru_maxrss and the way memory is accounted.
• The cgroup parser was exercised on invented sample text and has not read a real cgroup file.
• Timing values are from one machine and one run each. They illustrate how to measure and interpret, and should not be taken as typical performance.
• Nothing here covers privileged processes, container runtimes’ own behavior, or distribution-specific logging setups, each of which adds layers a platform team has to learn for its own environment.

The next article, Networking for Platform Engineers: DNS, TLS, Routing, and Timeouts, applies the same method to the network: a request that never arrives or arrives late has a path with named steps, and each step fails in its own way.

References

• Linux man-pages, signal(7)
• Linux man-pages, wait(2)
• Linux man-pages, getrlimit(2)
• Linux man-pages, cgroups(7)
• Linux man-pages, path_resolution(7)
• Linux man-pages, proc_pid_status(5)
• Linux man-pages, proc_pid_oom_score_adj(5)
• Linux kernel documentation, Control Group v2
• Linux kernel documentation, PSI – Pressure Stall Information
• Kubernetes documentation, Resource Management for Pods and Containers
• GNU Bash Reference Manual, Exit Status
• Python documentation, Command line and environment
• Python documentation, subprocess