Containerizing Data Jobs with Docker

Consider a hypothetical nightly job that reads the day’s order files, totals revenue per country, and writes a summary for a dashboard. It runs correctly on the developer’s laptop for weeks. When it moves to the scheduler host it fails on its first night, because the host’s Python is two minor versions older and a library resolved to a different release. A week later the same job succeeds but writes a summary a few cents different from a re-run, because someone upgraded the library on the host between the two runs. Nothing in the job’s code changed in either case.

Both failures come from the job’s environment being an accident of the machine it runs on. A container image removes that accident for the part of the environment that can be packaged: the interpreter, the libraries, the operating-system userland and the job’s own code. The rest of the environment (the input files, the configuration, the credentials, the CPU architecture, the memory the job may use) still has to be handled on purpose. This article builds one small batch job, packages it, and follows the decisions a data engineer has to make around it: how the image is built so that changes are cheap, what the container may read and write, how it stops, how its memory use is bounded, and how a release is named so that the exact bytes that ran can be found again.

The article assumes the project and packaging habits from Python for Data Engineers, including why a dependency lock file pins exact versions, and the process, signal and exit-code basics from Linux, the Shell, and Git for Data Work. How an image is stored as layers, and what namespaces and control groups do, is covered in Containers: Images, Isolation, and Resource Boundaries; this article uses those mechanisms and does not re-derive them. The Docker commands and the Dockerfile below are illustrative and were not executed, because no Docker daemon was available while writing. Everything labeled program output comes from Python programs that were run (Python 3.12 on macOS, standard library only) and that model the job’s behavior or the documented rules of the tooling.

What an image fixes and what it leaves open

A container image is a packaged root filesystem plus defaults for how to start it; a container is a running instance of it with its own writable layer and runtime settings. For a data job, the useful consequence is narrow and concrete: two machines that run the same image run the same interpreter, the same installed libraries and the same job code. That is the property the failure above lacked.

Four things stay outside the image, and each one corresponds to a decision later in the article. The input data is external, so identical code can still produce different results on different inputs. The configuration (which date to process, which directories to use) must be supplied when the container starts. Credentials must never be baked into the image. And a Linux container uses the host’s kernel and a specific CPU architecture, so an image built for one architecture does not run unchanged on another. Treating the image as “the environment” and forgetting these four is the usual source of surprise.

The job and its contract

The job below reads every CSV file in an input directory, sums amount_cents per country, and publishes one JSON summary into an output directory. The toy input is written by the demonstration script, and the job uses only the standard library so that it runs anywhere. What matters for containerization is the contract around the computation, which is written into the code deliberately.

• Configuration comes from environment variables: INPUT_DIR, OUTPUT_DIR and RUN_DATE. A container’s command line and environment are supplied at start, which makes the same image usable for any date and any mount layout.
• The exit status tells the scheduler what happened: 0 for success, 2 for bad configuration, 3 for bad input data, 73 for an output that cannot be written, and 143 when the job stopped because it received SIGTERM. A scheduler can only act on codes that the process actually returns; see exit status for the convention.
• Diagnostics go to standard error, which is where a container runtime collects logs without any configuration (the next sections show how).
• Output is published atomically: the job writes a temporary file in the output directory and renames it onto the final name, so a reader never sees a half-written summary. A re-run with the same date replaces the file with identical bytes, which is the idempotence a scheduler’s retry depends on.
• The summary records the names and checksums of the input files and an IMAGE_DIGEST value supplied from outside. The reason for taking the digest from outside is explained in the release section.

"""orders_job.py - a containerizable batch job (stdlib only).
Contract: read CSV files from INPUT_DIR, write one summary file into OUTPUT_DIR, exit 0 on success.
Exit codes: 0 ok, 2 bad configuration, 3 bad input data, 73 cannot write output, 143 stopped by SIGTERM."""
import csv, json, os, signal, sys, tempfile, hashlib
from pathlib import Path

stop_requested = False
def _on_term(signum, frame):
    global stop_requested
    stop_requested = True            # finish the current file, then stop cleanly
signal.signal(signal.SIGTERM, _on_term)

def main() -> int:
    try:
        in_dir = Path(os.environ["INPUT_DIR"]); out_dir = Path(os.environ["OUTPUT_DIR"])
        run_date = os.environ["RUN_DATE"]
    except KeyError as e:
        print(f"config error: missing {e.args[0]}", file=sys.stderr); return 2
    image_digest = os.environ.get("IMAGE_DIGEST", "unknown")   # injected by whoever starts the container
    totals, files = {}, []
    for f in sorted(in_dir.glob("*.csv")):
        if stop_requested:
            print("SIGTERM received: stopping before", f.name, file=sys.stderr); return 143
        with f.open(newline="") as fh:
            for row in csv.DictReader(fh):
                try:
                    totals[row["country"]] = totals.get(row["country"], 0) + int(row["amount_cents"])
                except (KeyError, ValueError):
                    print(f"bad row in {f.name}: {row}", file=sys.stderr); return 3
        files.append({"name": f.name, "sha256": hashlib.sha256(f.read_bytes()).hexdigest()[:12]})
        if os.environ.get("SLOW"): __import__("time").sleep(float(os.environ["SLOW"]))
    result = {"run_date": run_date, "totals_cents": totals, "inputs": files, "image": image_digest}
    try:
        out_dir.mkdir(parents=True, exist_ok=True)
        fd, tmp = tempfile.mkstemp(dir=out_dir, prefix=".tmp-")      # same filesystem as the target
        with os.fdopen(fd, "w") as fh: json.dump(result, fh, sort_keys=True)
        os.replace(tmp, out_dir / f"summary-{run_date}.json")        # atomic publish
    except OSError as e:
        print(f"cannot write output: {e.strerror} (errno {e.errno})", file=sys.stderr); return 73
    print(f"wrote summary-{run_date}.json", file=sys.stderr); return 0

if __name__ == "__main__":
    sys.exit(main())

The script demo1.py runs this job six times in a scratch directory with different conditions and prints the exit status and standard error of each. The last case starts a slow run, sends SIGTERM to it half a second in, and checks what is left in the output directory.

import os, subprocess, sys, signal, time, tempfile, stat
from pathlib import Path
JOB = Path("job.py").resolve()
root = Path(tempfile.mkdtemp()); inp = root/"in"; out = root/"out"; inp.mkdir()
(inp/"a.csv").write_text("country,amount_cents\nKR,1500\nUS,2500\n")
(inp/"b.csv").write_text("country,amount_cents\nKR,500\nDE,100\n")
base = dict(os.environ, INPUT_DIR="in", OUTPUT_DIR="out", RUN_DATE="2024-10-25", IMAGE_DIGEST="sha256:demo")
def run(label, env, **kw):
    p = subprocess.run([sys.executable, str(JOB)], env=env, cwd=root, capture_output=True, text=True, **kw)
    print(f"{label:28} exit={p.returncode} stderr={p.stderr.strip()!r}")
    return p
run("normal run", base); print("  output:", sorted(x.name for x in out.iterdir()))
before = (out/"summary-2024-10-25.json").read_bytes()
run("re-run same date", base); print("  identical bytes:", before == (out/"summary-2024-10-25.json").read_bytes())
e = dict(base); del e["RUN_DATE"]; run("missing RUN_DATE", e)
(inp/"c.csv").write_text("country,amount_cents\nFR,abc\n"); run("bad input row", base); (inp/"c.csv").unlink()
out.chmod(stat.S_IRUSR|stat.S_IXUSR); run("read-only output dir", dict(base, RUN_DATE="2024-10-26")); out.chmod(0o755)
# SIGTERM during a slow run
p = subprocess.Popen([sys.executable,str(JOB)], cwd=root, env=dict(base, RUN_DATE="2024-10-27", SLOW="1"), stderr=subprocess.PIPE, text=True)
time.sleep(0.5); p.send_signal(signal.SIGTERM); err = p.communicate()[1]
print(f"{'SIGTERM mid-run':28} exit={p.returncode} stderr={err.strip()!r}")
print("  output after SIGTERM:", sorted(x.name for x in out.iterdir()))

Program output:

normal run                   exit=0 stderr='wrote summary-2024-10-25.json'
  output: ['summary-2024-10-25.json']
re-run same date             exit=0 stderr='wrote summary-2024-10-25.json'
  identical bytes: True
missing RUN_DATE             exit=2 stderr='config error: missing RUN_DATE'
bad input row                exit=3 stderr="bad row in c.csv: {'country': 'FR', 'amount_cents': 'abc'}"
read-only output dir         exit=73 stderr='cannot write output: Permission denied (errno 13)'
SIGTERM mid-run              exit=143 stderr='SIGTERM received: stopping before b.csv'
  output after SIGTERM: ['summary-2024-10-25.json']

Each line maps to a decision a scheduler or an operator will make. The re-run with the same date leaves identical bytes, so a retry after an ambiguous failure is safe. A missing RUN_DATE is rejected with status 2 before any work is done, a malformed row stops the job with status 3 and the offending row in the message, and a non-writable output directory produces status 73 instead of a traceback with status 1. The SIGTERM case matters for containers specifically: the job finishes its current file, stops before starting the next one, returns 143 (128 plus signal number 15, the shell convention for death by SIGTERM) and leaves the output directory untouched, so no partial summary exists to be picked up downstream.

Writing the Dockerfile

A Dockerfile is a list of instructions; each filesystem-changing instruction produces a layer. Here is the file for the job. The instructions were not run, and the lock file it refers to is a hypothetical pin list (imagine a later version of the job that writes Parquet through pyarrow).

FROM python:3.12-slim
ENV PYTHONUNBUFFERED=1
RUN useradd --uid 10001 --no-create-home runner
WORKDIR /app
COPY requirements.lock ./
RUN pip install --no-cache-dir -r requirements.lock
COPY job.py ./
USER 10001
ENTRYPOINT ["python", "/app/job.py"]

Reading it from the top: FROM selects the base image, a Debian-based slim Python image. ENV PYTHONUNBUFFERED=1 makes Python flush its output immediately instead of buffering it, which keeps log lines from being lost or delayed when the process is killed. RUN useradd creates an unprivileged account with a fixed numeric ID. WORKDIR sets and creates the working directory. The two COPY and RUN pip install lines install dependencies from the lock file, COPY job.py adds the code, USER 10001 makes the container run as the unprivileged user by default, and ENTRYPOINT in exec form (a JSON array) names the process to run. The user ID, the exec form and the install order each have a reason that the following subsections give.

Why the order of instructions matters

Docker’s build cache reuses a layer when the instruction and its inputs are unchanged. The documentation states how it decides: for RUN, only the command text is compared, not what the command would produce; for COPY and ADD, a checksum of the copied files is compared and the modification time is ignored; and once one step misses, every later step is rebuilt. These three rules explain the order above. Dependencies change rarely and take the longest to install, and the job’s code changes constantly, so the lock file is copied and installed first and the code last. A code edit then invalidates only the final steps.

A short Python program applies exactly these three rules to a Dockerfile and a build-context directory, and reports which instructions would be re-executed after each change. It is a model of the documented rule, not Docker: it hashes instruction text and file content and chains each key to its parent’s key. It is run once on the Dockerfile above and once on a variant that copies the code and the lock file together before installing (Dockerfile.naive).

import hashlib, shutil, sys
from pathlib import Path

def h(*parts):
    return hashlib.sha256("\0".join(parts).encode()).hexdigest()[:10]

def build(dockerfile, ctx, cache):
    """Return the instructions that had to be re-executed (cache misses)."""
    parent, misses = "", []
    for line in dockerfile.read_text().splitlines():
        op, _, arg = line.partition(" ")
        if op == "COPY":
            srcs = arg.split()[:-1]
            content = "".join(h((ctx / s).read_text()) for s in srcs)  # file content, not mtime
            key = h(parent, line, content)
        else:
            key = h(parent, line)                                      # command text only
        if key not in cache:
            misses.append(line)
        cache.add(key)
        parent = key                                                   # a miss changes every later key
    return misses

ctx, cache = Path("ctx"), set()
shutil.copy("job.py", ctx / "job.py")
(ctx / "requirements.lock").write_text("pyarrow==17.0.0\n")
dockerfile = Path(sys.argv[1])

def step(title):
    m = build(dockerfile, ctx, cache)
    print(f"{title}: {len(m)} step(s) re-executed")
    for line in m:
        print("   ", line)

step("1 first build")
step("2 nothing changed")
(ctx / "job.py").write_text((ctx / "job.py").read_text() + "\n# tweak\n")
step("3 edit job.py")
(ctx / "requirements.lock").write_text("pyarrow==17.0.0\nnumpy==2.0.2\n")
step("4 edit requirements.lock")

Program output for the Dockerfile above:

1 first build: 9 step(s) re-executed
    FROM python:3.12-slim
    ENV PYTHONUNBUFFERED=1
    RUN useradd --uid 10001 --no-create-home runner
    WORKDIR /app
    COPY requirements.lock ./
    RUN pip install --no-cache-dir -r requirements.lock
    COPY job.py ./
    USER 10001
    ENTRYPOINT ["python", "/app/job.py"]
2 nothing changed: 0 step(s) re-executed
3 edit job.py: 3 step(s) re-executed
    COPY job.py ./
    USER 10001
    ENTRYPOINT ["python", "/app/job.py"]
4 edit requirements.lock: 5 step(s) re-executed
    COPY requirements.lock ./
    RUN pip install --no-cache-dir -r requirements.lock
    COPY job.py ./
    USER 10001
    ENTRYPOINT ["python", "/app/job.py"]

Program output for the variant that copies job.py and requirements.lock in one COPY before the install step:

1 first build: 8 step(s) re-executed
    FROM python:3.12-slim
    ENV PYTHONUNBUFFERED=1
    RUN useradd --uid 10001 --no-create-home runner
    WORKDIR /app
    COPY job.py requirements.lock ./
    RUN pip install --no-cache-dir -r requirements.lock
    USER 10001
    ENTRYPOINT ["python", "/app/job.py"]
2 nothing changed: 0 step(s) re-executed
3 edit job.py: 4 step(s) re-executed
    COPY job.py requirements.lock ./
    RUN pip install --no-cache-dir -r requirements.lock
    USER 10001
    ENTRYPOINT ["python", "/app/job.py"]
4 edit requirements.lock: 4 step(s) re-executed
    COPY job.py requirements.lock ./
    RUN pip install --no-cache-dir -r requirements.lock
    USER 10001
    ENTRYPOINT ["python", "/app/job.py"]

In the ordered file, a code edit re-executes three cheap steps and does not reinstall dependencies. In the variant, the same edit changes the checksum of the shared COPY, which invalidates the pip install that follows, so every code change pays for a full dependency install. Editing the lock file correctly rebuilds the install in both files. The cost of the mistake grows with the install time; for a job with a large scientific stack, the difference is between seconds and minutes per build.

The rule for RUN also has a trade-off in the other direction. Because only the command text is compared, RUN pip install -r requirements.lock is a cache hit a month later even though a newer base image or a package index might now give a different result for anything not pinned. Docker’s documentation says to clear the cache or build with --no-cache when you want updated packages. For a data job the safer reading is that the lock file, not the cache, defines the dependencies: with exact pins and hashes (pip’s hash-checking mode), a cached layer and a fresh install produce the same libraries, and an unpinned dependency is the bug to fix.

The build context and what stays out of the image

docker build sends a directory, the build context, to the builder, and COPY can only read from it. A .dockerignore file excludes paths from the context in the same style as .gitignore. For a data repository it should exclude data directories, notebooks’ output, .git, local virtual environments and any file that holds credentials. Excluding data is a size matter, because a context with gigabytes of sample files slows every build, and a correctness matter, because a stray COPY . . would otherwise bake a day’s data into the image and make the image look reproducible while carrying hidden inputs. Layers are permanent: a file added in one layer and deleted in a later one is hidden, not removed, and remains readable by anyone who can pull the image; the earlier article on container images shows this with a byte-level example.

Base image and user

Pin the base image to a specific version rather than a bare name such as python, which moves. A slim Debian-based Python image is a common default for data libraries because prebuilt binary wheels for glibc-based Linux are widely published; images based on a different C library may need those packages to be compiled or may lack a matching wheel, which turns a simple install into a build problem. Check the wheels for your dependency set before choosing a smaller base image.

The numeric USER 10001 is deliberate. File ownership on Linux is stored as numbers, so when a bind mount exposes a host directory, a container process running as user 10001 is subject to the host’s permissions for the numeric ID 10001, regardless of what the image calls that user. Running as root would sidestep permission errors on the mount, and it would also let a bug or an unreviewed dependency write anywhere the mount allows. The job’s status 73 in the demonstration is how such a permission problem should surface: one line and a distinct exit code. How strictly Docker Desktop on macOS or Windows maps ownership across its file-sharing layer differs from a Linux host, so verify the permission behavior on the platform where the job runs.

The container lifecycle, applied

The image is built once; containers are created from it as often as needed. The commands are listed below as a sequence a scheduler would perform for each run (not executed here).

docker build -t registry.example/orders-job:2024.10 .
docker create --name orders-2024-10-25 \
    -e INPUT_DIR=/data/in -e OUTPUT_DIR=/data/out -e RUN_DATE=2024-10-25 \
    registry.example/orders-job:2024.10
docker start -a orders-2024-10-25      # run in the foreground; the exit status is the job's
docker stop orders-2024-10-25          # only needed if it is still running
docker rm orders-2024-10-25            # discards the container's writable layer

docker run combines create and start, and --rm adds the removal when the process exits. The distinction between stopped and removed is where beginners lose data. A stopped container still exists with its writable layer and can be inspected; removal discards that layer. Anything the job wrote to a path that is neither in the image nor on a mount lives only in that layer.

The next program models the rules stated in the container image and container entry: an image is read-only, each container has its own writable layer, a mount is a directory outside the container, and removal discards only the writable layer. It is a toy built from temporary directories, not a container runtime.

"""Toy container lifecycle: an image is read-only; each container gets its own writable layer
(a temp dir) that disappears on remove; a mount is a directory that outlives containers."""
import shutil, tempfile
from pathlib import Path

class Image:
    def __init__(self, files): self.files = dict(files)            # read-only
class Container:
    def __init__(self, image, mounts):
        self.image, self.upper = image, Path(tempfile.mkdtemp(prefix="upper-"))
        self.mounts = mounts                                       # container path -> host dir
    def write(self, path, text):
        for dst, host in self.mounts.items():
            if path.startswith(dst + "/"):
                (Path(host) / path[len(dst) + 1:]).write_text(text); return "mount"
        (self.upper / path.strip("/").replace("/", "__")).write_text(text); return "writable layer"
    def remove(self): shutil.rmtree(self.upper)                    # discards the writable layer only

results = Path(tempfile.mkdtemp(prefix="results-"))               # stands in for a volume or bind mount
img = Image({"/app/job.py": "print('v1')"})
c1 = Container(img, {"/out": results}); c2 = Container(img, {"/out": results})
print("c1 writes /out/summary.json   ->", c1.write("/out/summary.json", "{}"))
print("c1 writes /tmp/scratch.txt    ->", c1.write("/tmp/scratch.txt", "x"))
print("c1 edits /app/job.py          ->", c1.write("/app/job.py", "print('hotfix')"))
print("image still says:", img.files["/app/job.py"], "| c2 sees image file:", c2.image.files["/app/job.py"])
print("c1 upper layer before remove:", sorted(p.name for p in c1.upper.iterdir()))
c1.remove()
print("c1 removed; upper layer exists:", c1.upper.exists())
print("results dir after remove:", sorted(p.name for p in results.iterdir()))

Program output:

c1 writes /out/summary.json   -> mount
c1 writes /tmp/scratch.txt    -> writable layer
c1 edits /app/job.py          -> writable layer
image still says: print('v1') | c2 sees image file: print('v1')
c1 upper layer before remove: ['app__job.py', 'tmp__scratch.txt']
c1 removed; upper layer exists: False
results dir after remove: ['summary.json']

Three facts follow from the output. The “hotfix” written inside container c1 exists only in c1‘s writable layer; the image still contains the original job.py and a second container created from it sees the original. To ship a change, edit the source, rebuild and release a new image. The scratch file written under /tmp disappeared with the container, which is fine for scratch and fatal for results. And the summary written under the mounted path survived the removal, because the mount lives outside the container’s lifecycle. A related command, docker commit, can turn a container’s state into an image, but the result is an image nobody can rebuild from source, so it is not a release mechanism for data jobs.

Reading inputs and publishing results: mounts

A job needs to see input files and leave results somewhere that outlives it. Docker provides three mechanisms: a bind mount exposes a directory of the host that runs the Docker daemon, a named volume is storage that Docker manages, and a tmpfs mount is memory-backed scratch. The choices below follow from the roles the data plays.

# Reviewed inputs on a known host: read-only. Results: a separate, writable mount.
docker run --rm \
    --mount type=bind,source=/srv/orders/incoming/2024-10-25,target=/data/in,readonly \
    --mount type=bind,source=/srv/orders/summaries,target=/data/out \
    -e INPUT_DIR=/data/in -e OUTPUT_DIR=/data/out -e RUN_DATE=2024-10-25 \
    registry.example/orders-job:2024.10

The input is mounted read-only so that a bug in the job cannot damage the source files, and the results go to a different mount so that write access is granted to exactly one place. Two behaviors of bind mounts are worth knowing before they surprise someone. With -v, a nonexistent host path is created for you as a directory, so a typo in the source path silently produces an empty input; --mount fails with an error by default, which is the better behavior for a job that should fail loudly. And a mount hides whatever the image already had at the target path; if the job’s code lived under /data, mounting there would replace it with the host’s directory.

A bind mount’s source is a path on the host that runs the Docker daemon. With a remote daemon, the path names a location on that remote machine, not on the laptop that typed the command, which is a frequent source of “my files are not there” in CI. Docker Desktop handles the translation for local use; a remote daemon does not.

When the job needs state that persists between runs but should not depend on a host path, such as a watermark file or a batch load ledger the job maintains, a named volume fits. It is created and referenced by name, and any replacement container can attach it on the same host.

docker volume create orders-state
docker run --rm --mount type=volume,source=orders-state,target=/data/state \
    -e STATE_DIR=/data/state registry.example/orders-job:2024.10

Persistence is not protection. Removing a container does not remove its named volume, and an anonymous volume created without --rm also survives, so volumes accumulate until someone removes them with docker volume rm or docker volume prune. The prune command deletes volumes that no container uses, which is the wrong thing to run on a host where a stopped container’s state is the only copy. A volume is also local to its host unless a volume driver provides remote storage, so a job rescheduled onto another machine does not find it. Docker’s documentation describes backing a volume up by starting a throwaway container that mounts it next to a host directory and writes a tar archive; the backup schedule, retention and restore test are decisions the job’s owner makes, separately from how the job is deployed.

One interaction between mounts and the job’s contract deserves care. The job renames a temporary file onto the final name to publish atomically, and a rename is atomic only within one filesystem. The temporary file is created inside the output directory for that reason; creating it in the container’s /tmp and renaming across the mount boundary would fail with a cross-device error, and a copy-then-delete fallback could be observed half-written. See atomic publication for the reader-side reasoning. A bind mount that crosses a network filesystem has weaker rename guarantees, which is a reason to treat the published file’s name, not its existence, as the commit marker and to record completion separately.

Stopping a container cleanly

docker stop sends SIGTERM to the container’s main process and, if it has not exited after a grace period (10 seconds by default on Linux), sends SIGKILL; the signal and the period can be changed with the --signal and --timeout options, and an image can set its default signal with STOPSIGNAL. An orchestrator replacing a task or draining a node uses the same sequence. A job that ignores SIGTERM does not stop; it is killed ten seconds later, mid-write.

Whether SIGTERM reaches the job depends on how the Dockerfile starts it. In shell form, ENTRYPOINT python /app/job.py runs the command as a child of /bin/sh -c, so the shell is the container’s first process and the Python process does not receive the signal directly. In exec form, ENTRYPOINT ["python", "/app/job.py"], Python is that first process. The first process of a PID namespace has special signal semantics that the earlier article covers: signals are delivered to it only if it installed a handler. This is why the job installs one, as demo1.py exercised: it sets a flag, finishes the current file and returns 143 from a point where no partial output exists. A job that does a long single operation, such as one huge query, has to check the flag in a loop it controls or accept being killed, and its design should make that kill safe through the atomic publish and idempotent re-run shown above.

Memory: the limit is part of the job’s design

A container can be given a memory limit with --memory. When the process exceeds it, the kernel’s out-of-memory killer terminates it, the container exits (by the shell convention with status 137, 128 plus signal 9), and the job usually leaves no message because it never got the chance to write one. Docker records the event in the container’s state, which docker inspect shows, so the first diagnostic step is to look there before suspecting the code. Cgroup accounting, including what counts toward the limit, is explained in the earlier article; the data-engineering consequence is that a job’s peak memory must be known and bounded, not discovered when the scheduler moves it to a smaller host.

The program below makes the design lever visible. It sums a synthetic 200,000-row CSV (about 1.6 MB, written by the script) in two ways and measures peak Python heap with tracemalloc: one materializes every row before summing, the other streams rows and keeps only the per-country totals.

"""Peak Python heap while summing a CSV: read everything first vs stream rows (tracemalloc).
Synthetic data: 200,000 rows written by this script."""
import csv, tempfile, tracemalloc, os
from pathlib import Path
p = Path(tempfile.mkdtemp()) / "big.csv"
with p.open("w", newline="") as f:
    w = csv.writer(f); w.writerow(["country", "amount_cents"])
    for i in range(200_000): w.writerow([("KR","US","DE")[i % 3], i % 1000])
def load_all():
    with p.open(newline="") as f: rows = list(csv.DictReader(f))
    t = {}
    for r in rows: t[r["country"]] = t.get(r["country"], 0) + int(r["amount_cents"])
    return t
def stream():
    t = {}
    with p.open(newline="") as f:
        for r in csv.DictReader(f): t[r["country"]] = t.get(r["country"], 0) + int(r["amount_cents"])
    return t
print("file size: %.1f MB" % (os.path.getsize(p) / 1e6))
for fn in (load_all, stream):
    tracemalloc.start(); res = fn(); _, peak = tracemalloc.get_traced_memory(); tracemalloc.stop()
    print(f"{fn.__name__:9} peak heap {peak/1e3:9.1f} KB  result={res}")

Program output:

file size: 1.6 MB
load_all  peak heap   55748.4 KB  result={'KR': 33300333, 'US': 33300000, 'DE': 33299667}
stream    peak heap      47.9 KB  result={'KR': 33300333, 'US': 33300000, 'DE': 33299667}

Both functions return the same totals, yet the first peaks at about 56 MB of Python heap for a 1.6 MB file, and the second stays near 48 KB. The first one’s memory grows with the input, so a larger day or a backfill over a month would exceed any fixed limit; the second one’s memory depends on the number of distinct countries, not rows. Parsed Python objects take much more memory than the bytes on disk, which is why file size is a poor predictor of memory use. For a containerized job, choose the streaming or chunked form first, then set a limit with headroom above the measured peak of a representative large run, and keep the measurement in the job’s documentation. Libraries that process data out of memory, such as the ones in Single-Node Analytics: DuckDB, Polars, and Arrow, move the same lever inside the engine.

Configuration, secrets, and logs

The job reads its settings from the environment because an image is a shared, content-addressed artifact: the same image is promoted from test to production, and only the settings differ (promoting the artifact that was tested is the topic of the CI/CD article). Settings are passed with -e or --env-file at run time. Validate them at startup so that a missing or malformed value stops the job before it does work, as RUN_DATE does above; the configuration validation entry describes the pattern.

Credentials need more care than ordinary settings. A value set with ENV in the Dockerfile persists in the image’s configuration, and a build argument that was used during the build can persist in the image’s history too, so neither is a place for a secret. Supply secrets when the container starts, from the scheduler’s or platform’s secret store, or use the build tool’s dedicated secret mount for build-time needs, which keeps the bytes out of the layers. Treat any secret that ever reached a published layer as disclosed and rotate it. The secrets management entry and the earlier image article cover the mechanism.

Write logs to standard output or standard error. The runtime collects both streams from the container’s main process, docker logs reads them, and a scheduler or log shipper can forward them without the job knowing where they go. A job that writes its own log file inside the container loses it when the container is removed. One event per line with stable field names, as in structured logging, makes the run searchable; the run date, the input names and the image reference are the fields that most often answer “which run was this”.

Naming a release: tag and digest

After the image is built and tested, it is pushed to a registry and referenced by name. A tag such as 2024.10 is a pointer that the registry’s owner can move: pushing a different image under the same tag makes the name resolve to new content. A digest, written repository@sha256:..., is the SHA-256 of the image’s manifest and names exact content. The tag and digest entry gives the definitions; the following program models the behavior with a toy registry whose digests are real SHA-256 values of manifest bytes that the script constructs.

"""Toy registry: a tag is a movable pointer; a digest is sha256 of the manifest bytes."""
import hashlib, json
def manifest(layers, platform="linux/amd64"):
    return json.dumps({"layers": layers, "platform": platform}, sort_keys=True).encode()
def digest(m): return "sha256:" + hashlib.sha256(m).hexdigest()
blobs, tags = {}, {}
def push(repo, tag, m):
    d = digest(m); blobs[(repo, d)] = m; tags[(repo, tag)] = d; return d
def resolve(ref):
    if "@" in ref: repo, d = ref.split("@"); assert (repo, d) in blobs; return d
    repo, tag = ref.split(":"); return tags[(repo, tag)]

r = "registry.example/orders-job"
d1 = push(r, "2024.10", manifest(["base-3.12.6", "deps-A", "code-1"]))
print("release record :", f"{r}@{d1[:19]}...")
print("tag resolves to:", resolve(f"{r}:2024.10")[:19], "...")
d2 = push(r, "2024.10", manifest(["base-3.12.7", "deps-A", "code-1"]))   # patched base, same tag
print("after re-push, tag :", resolve(f"{r}:2024.10")[:19], "... changed:", resolve(f"{r}:2024.10") != d1)
print("pinned reference   :", resolve(f"{r}@{d1}")[:19], "... changed:", resolve(f"{r}@{d1}") != d1)
d3 = push(r, "2024.10-arm", manifest(["base-3.12.6", "deps-A", "code-1"], platform="linux/arm64"))
print("same layers, other platform -> same digest?", d3 == d1)

Program output:

release record : registry.example/orders-job@sha256:1f4109d3fe00...
tag resolves to: sha256:1f4109d3fe00 ...
after re-push, tag : sha256:9f7153c61ee3 ... changed: True
pinned reference   : sha256:1f4109d3fe00 ... changed: False
same layers, other platform -> same digest? False

The release record holds the digest. After a second push to the same tag (here a patched base layer), the tag resolves to new content, while the digest reference still resolves to the original. A job started by tag on Monday and again on Tuesday could therefore run different code under an unchanged name, which is the original failure in a new place. The last line shows that the same layers built for another platform produce a different digest, because the manifest differs; with a multi-platform image, the name resolves through an index to a per-platform manifest, so the release record should keep the intended operating system and architecture with the reference, and the digest recorded after docker push (the push output prints it) is the one to store.

Starting the job from the digest looks like docker run registry.example/orders-job@sha256:<digest>. A container cannot read its own image digest from inside, because the digest is computed from content that already includes the image’s files, so including it in a file would change the digest. The pattern that works is to let the party that starts the container pass it in, as the IMAGE_DIGEST variable the job copies into its summary. A reader of an output file can then tell exactly which image produced it, next to the input checksums the job also recorded. Together, the image digest, the input checksums and the settings are the run’s reproducibility record; the image alone does not reproduce a result if the inputs changed. The article on reproducible service artifacts discusses the build-side controls (locks, deterministic builds, provenance) that make the digest meaningful.

Pinning has a price, which the callers’ glossary entries also state. When a base-image maintainer patches a vulnerability behind the tag, a pinned digest does not pick the patch up. Updates become an explicit process: rebuild on a schedule or on advisory, run the job’s tests against the new image, promote the new digest, record it, and keep the previous digest until the new one has run successfully. The same applies to FROM python@sha256:... in the Dockerfile, which makes the base itself deterministic at the cost of the same manual update. A digest identifies content; it does not prove that the content is trustworthy or free of vulnerabilities, and it does not freeze the data the job reads.

Architecture is the other release detail. An image built on an Apple-silicon laptop is, by default, built for that machine’s architecture and may not start on an x86-64 scheduler host, where the failure appears as an exec format error or, with emulation, as slow runs. Build for the platforms you deploy (docker build --platform, or a multi-platform build) and record which one the release record’s digest points to.

Failure modes and where each is checked

The decisions above turn vague failures into specific ones. The table collects the symptoms a data job shows in practice, the usual cause, and the first thing to inspect.

Symptom	Usual cause	First check
Job exits immediately with status 126 or 127	The command in the image cannot be run or does not exist (a wrong entrypoint path, a missing executable bit)	The container’s start command, the image’s platform, and the path in ENTRYPOINT
Exit status 137 and no error message	The memory limit was exceeded and the kernel killed the process	The container’s OOM-killed state, the limit, and the job’s peak memory on a representative run
Output exists but is empty or from an old date	A mount hides the intended directory, or -v created an empty host directory	The mount list of the container and the host path on the daemon’s machine
Permission denied writing results	The container user’s numeric ID has no write access to the host directory	The numeric owner and mode of the host directory against the image’s USER
Results disappear after a run	The job wrote to the container’s writable layer and the container was removed	Which paths the job writes to, and whether each is on a mount
Same tag, different results	The tag moved to a new image	The digest recorded in the output, against the release record
Job is killed ten seconds after a stop request	The process did not handle SIGTERM, often because a shell is PID 1	Exec-form ENTRYPOINT and the job’s signal handler
Every code change reinstalls dependencies	Code is copied before the install step	The order of COPY and RUN in the Dockerfile

Where a container job runs next

A scheduler such as Apache Airflow can run a container as a task, for example through the Docker provider’s operator on a worker host or by launching a Kubernetes pod; in either form the contract built above is what the scheduler consumes: environment in, a defined exit status out, logs on the standard streams, and results on a mount or in object storage. Moving from one Docker host to a cluster adds scheduling, resource requests and limits, and storage that survives a node change, which the next article, Running Data Workloads on Kubernetes, takes up. Neither runtime removes the job’s own responsibility for idempotent re-runs: containers make the environment repeatable, and the job’s design makes its effects repeatable.

A container is also not always the right unit. A job that is a single SQL statement against a warehouse gains little from packaging, and a long-lived service has a different lifecycle from a batch job. The pattern earns its keep when the job has real dependencies, runs on machines the author does not control, and needs a version that can be named, promoted and rolled back.

References

• Docker: Dockerfile reference
• Docker: Build cache invalidation
• Docker: Bind mounts
• Docker: Volumes
• Docker: docker container stop
• Docker: Running containers
• Docker: docker image push
• Open Container Initiative: Image Format Specification
• Containers: Images, Isolation, and Resource Boundaries
• Packaging Services and Building Reproducible Artifacts