Linux, the Shell, and Git for Data Work

In plain terms

A pipeline may run on a laptop, a remote server, or a short-lived container. When there is no graphical interface, commands let you inspect files, check a running process, and read the evidence it leaves behind.

The terminal is the text interface; the shell is the program that interprets commands entered there. Linux is the operating-system environment, Bash is one shell, and Git is a separate version-control tool. Shells can run commands sequentially, in pipelines, or in the background. Web consoles can expose similar information, but shell tools make many inspections easy to combine and repeat.

The second half uses Git to record proposed changes, combine work through review, and investigate which code changed. Connecting a commit to an incident also requires knowing what was deployed and what data it processed.

The examples use Bash, including on macOS where the interactive default may be zsh. Generate the synthetic 100,000-line log in Lab 1 before running Part 2. Linux-specific paths, account administration, and SSH labs are marked separately; commands with remote hosts or system paths are examples to adapt, not a script to paste wholesale.

Part 1: the machine

Files, paths, and where things live

Linux organises everything as one tree starting at /; there are no drive letters. A few locations come up constantly:

A path starting with / is absolute within the current filesystem namespace; the same spelling may refer to different files in another container or machine. One without it is relative to the current directory. Scheduled jobs fail on this distinction: a script that works when run from its own folder breaks when the scheduler runs it from /, because config.yaml is no longer “right here”. A script that will be run by something other than you should use absolute paths or compute its own location first.

pwd                 # current directory
ls -la              # contents, including hidden files and permissions
cd /var/log         # go there
cd -                # go back to the previous directory
mkdir -p a/b/c      # create nested folders; no error if they exist
cp -r src dst       # copy a folder
mv old new          # rename or move (same operation)
rm -ri -- folder        # inspect and confirm deletions; no trash restore

rm -r removes directories recursively; it does not provide a trash restore. The illustrative command uses -i to prompt and -- to end options. Confirm the exact target and keep destructive practice inside disposable scratch data. Quoting protects argument boundaries; it does not make an unintended path correct.

Permissions

Ordinary Unix permissions have read, write, and execute bits for the owner, group, and others. Additional ACLs and security policies can also affect access. For a directory, read lists names, execute permits traversing it, and write together with execute permits changing its entries. Reading a file requires traversal permission on every parent directory as well as file access. The Linux path-resolution manual explains directory search checks. ls -l shows the basic bits:

-rwxr-x---  1 airflow  data   4096 Mar 14 09:00 run_etl.sh
 │└┬┘└┬┘└┬┘   │        │
 │ │  │  │    owner    group
 │ │  │  └── others: nothing
 │ │  └───── group "data": read + execute
 │ └──────── owner "airflow": read + write + execute
 └────────── "-" = regular file, "d" = directory, "l" = link

A scheduler may run as a different account from the one that created the input. Check the actual user and groups with id, the file’s owner and mode, and parent-directory access. These commands illustrate distinct changes; apply only the one justified by the intended access:

chmod 640 file        # owner rw, group r, others nothing (r=4 w=2 x=1)
chmod u+x script.sh    # make it runnable
chown airflow:data f  # change owner and group

sudo runs an authorized command as another user, root by default. chmod u+x adds owner execution; it does not fix missing directory access or ownership. Grant the scheduler’s user or group the access it needs rather than using 777 as a blanket response.

Processes, signals, and exit codes

A running program is a process with a numeric ID (PID).

ps aux | grep '[s]park'      # find processes by name
top                      # live CPU and memory; q to quit (htop is easier to read)
kill 12345               # ask process 12345 to stop (SIGTERM)
kill -9 12345            # force it (SIGKILL): no cleanup, no flush; last resort
nohup ./long_job.sh &    # run in the background and ignore SIGHUP; no restart supervision

SIGTERM gives a program an opportunity to handle termination; cleanup only happens if the program implements it. SIGKILL cannot be caught and prevents application cleanup. Confirm the process identity before signaling it, allow a grace period, and check its exit and outputs. Neither signal guarantees that previously written data is complete.

When a process ends the shell observes its exit status: 0 for success, non-zero for an unsuccessful or special result according to the command’s contract. Schedulers, CI systems, and orchestrators use this value to decide whether a step worked. A script that prints “ERROR: load failed” and then exits with 0 has reported success, so the next step runs on missing data. Check a code with echo $? right after a command, and in your own scripts make failure exit non-zero. Lab 3 demonstrates why printing an error and reporting failure are separate actions. Save status=$? immediately if you need the value later: another command replaces it. A successful exit reports the command’s contract, not necessarily complete or correct business data.

Environment variables and secrets

An environment variable is a named value a process inherits from whatever started it. PATH lists the folders searched for commands; HOME is the home directory. Pipelines use environment variables for configuration that differs between environments: database hosts, bucket names, and credentials. A new shell variable stays local unless exported; export includes it in subsequently launched commands. A child’s change does not update its parent or an already running job. A scheduler may therefore see different values from your terminal.

export DB_HOST=warehouse.internal   # non-secret configuration
printf '%s\n' "$DB_HOST"             # inspect this value, not the whole environment

Keep credentials out of committed code and configuration. A runtime secret store or injected environment can deliver them, but environment variables are not a vault: debug output, inherited processes, or diagnostic dumps may expose them. Avoid printing the whole environment when inspecting a job. If a credential is committed, revoke or rotate it promptly; deleting the file does not invalidate the credential.

Part 2: streams and pipes

Commands normally start with three standard streams: standard input (stdin, file descriptor 0), standard output (stdout, descriptor 1), and standard error (stderr, descriptor 2). These descriptors can point to a terminal, file, or pipe, and a program can close or replace them. By convention, stdout carries results and stderr carries diagnostics, including progress messages. Text on stderr does not itself mean that the command failed; inspect its exit status separately.

command > out.txt        # stdout to a file (overwrite)
command >> out.txt       # append
command 2> err.txt       # stderr to a file
command > all.txt 2>&1   # both to one file; the usual form for job logs
command1 | command2      # stdout of 1 becomes stdin of 2: a pipe

command < in.txt supplies a file as stdin. Redirections are applied left to right: > all.txt 2>&1 sends both outputs to the file, but reversing those redirections can leave stderr on the terminal. The shell opens and truncates a > destination before running the command, so never transform a file by redirecting output back onto that same input path. An ordinary | connects stdout only; stderr keeps its existing destination. A pipe carries bytes, while these particular tools interpret those bytes as lines. Without pipefail, a foreground pipeline reports its last command’s status, so a successful final tool can hide an earlier failure.

Each tool does one small thing to a stream of lines, and pipes chain them. The vocabulary:

Worked examples from a bad morning

The log below has one line per event: a timestamp, a level, the worker that wrote it, and a message. Each line looks like this:

head -n 3 app.log
# 2026-03-14T00:00:00 INFO worker-1 checkpoint written
# 2026-03-14T00:00:00 INFO worker-8 batch committed
# 2026-03-14T00:00:01 INFO worker-8 heartbeat ok

How many errors, and which kinds? In this generated log, ERROR occurs only in the level field; in other logs use a field-aware filter such as awk '$2 == "ERROR"' to avoid matching message text. Count the matching lines, then keep only the message (field 4 onwards, splitting on spaces), sort so that identical messages sit together, collapse and count them, and sort the counts:

grep -c ERROR app.log
# 3625
grep ERROR app.log | cut -d' ' -f4- | sort | uniq -c | sort -rn | head
# 2179 connection refused to warehouse.internal:5439
#  898 timeout reading from vendor api
#  343 schema mismatch in orders.csv column 7
#  205 out of disk space on /tmp

About 60% of the errors are one message, which is where to look first. The sort | uniq -c pair is the shell’s GROUP BY; uniq only merges neighbours, which is why the sort comes first.

When did it start going wrong? The first 13 characters of each line are the date and hour. Group on those:

grep ERROR app.log | cut -c1-13 | sort | uniq -c | head -n 5
#   80 2026-03-14T00
#   81 2026-03-14T01
#   82 2026-03-14T02
# 1680 2026-03-14T03
#   78 2026-03-14T04

This synthetic log has a spike in the 03:00 hour. Hourly counts locate an interval, not the exact incident start or a normal baseline for a real service. Compare error rates and inspect finer timestamps next. Which workers? awk can filter on field 2 and count by field 3:

awk '$2 == "ERROR" {n[$3]++} END {for (w in n) print w, n[w]}' app.log | sort
# worker-1 433
# worker-2 492
# worker-3 448
# worker-4 429
# worker-5 470
# worker-6 463
# worker-7 454
# worker-8 436

Similar counts across workers suggest checking shared dependencies, but they do not establish a common cause. Compare each worker’s error rate against its total workload and inspect timing and message types before attributing the failure to the warehouse.

The disk is full. What is using it? These depend on the machine, so no output is shown:

df -h                              # which filesystem is at 100%
du -sh /data/* | sort -rh | head   # largest folders under /data
find /tmp -size +1G -mtime +7      # large files in /tmp older than a week

Did yesterday’s files arrive, and how many rows?

ls -la /landing/orders/2026-03-13/
wc -l /landing/orders/2026-03-13/*.csv
awk -F, 'NR>1 {s+=$7} END {print s}' /landing/orders/2026-03-13/part-0.csv   # sum column 7, skip the header

What did the API return? Report HTTP status and elapsed time separately from the command’s exit status. The output comments below are hypothetical responses, not measurements of the placeholder host. A 200 status alone does not validate the response data; a 503 does not identify the responsible component.

curl --fail --silent --show-error --connect-timeout 5 --max-time 30 -o /dev/null -w "%{http_code} %{time_total}s\n" https://vendor.example/v1/orders
# 200 0.412s  # illustrative HTTP response; validate body separately
# 503 0.820s  # illustrative HTTP error; --fail returns a non-zero exit status

These commands are inspections, not a completeness proof. Compare received files against a manifest and parse the actual data format before accepting a delivery. wc -l counts newline characters, not CSV records; a quoted newline can make one record span multiple physical lines, and cut -d, or awk -F, does not implement CSV quoting. Use a CSV parser for general CSV.

Part 3: reaching other machines

SSH can provide an encrypted remote shell. Public-key authentication is one option: protect the private key and install the public key for the intended remote account. Server host-key verification is a separate step; confirm an unfamiliar fingerprint through a trusted channel instead of blindly accepting it. The OpenSSH manual describes authentication and host-key handling. The examples below require your own authorized host.

ssh-keygen -t ed25519                      # make a key pair, once
ssh user@10.0.3.7                          # connect
scp report.csv user@host:/data/inbox/      # copy a file there
rsync -avz --partial --progress /local/dir/ user@host:/remote/dir/   # copy contents; retain partial data for a later run
ssh -N -L 127.0.0.1:8080:warehouse.internal:5439 user@bastion   # tunnel: local port 8080 to the warehouse, via a jump host

Save host aliases in ~/.ssh/config. For interrupted transfers, rsync --partial keeps partial data for use on a later invocation; it does not retry forever by itself. By default, rsync selects changed files using size and modification time, not a full content hash. A source trailing slash copies directory contents. See the rsync manual for these options and deletion semantics.

tmux keeps a terminal session on the server when a client disconnects, provided the server and tmux process remain alive. It is not a scheduler, durable log store, or protection against reboot. Without tmux, a job’s survival depends on signals, shell settings, and redirection; nohup handles hangup but does not supervise or restart a job.

Part 4: scripts that fail safely

A script makes repeated commands reviewable. This Bash example copies a dated input directory into a dedicated destination and reports a failed copy. Supply absolute source and destination roots you control; keep them separate, with a fixed input snapshot and one writer. It illustrates error handling, not atomic dataset publication:

#!/usr/bin/env bash
set -euo pipefail

RUN_DATE="${1:?usage: $0 YYYY-MM-DD SOURCE_ROOT DEST_ROOT}"
SOURCE_ROOT="${2:?absolute source root required}"
DEST_ROOT="${3:?absolute destination root required}"
log() { printf '%s %s\n' "$(date -u +%FT%TZ)" "$*" >&2; }

[[ "$RUN_DATE" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]] || { log "bad date format"; exit 2; }
[[ "$SOURCE_ROOT" = /* && "$DEST_ROOT" = /* ]] || { log "roots must be absolute"; exit 2; }
SRC="${SOURCE_ROOT%/}/${RUN_DATE}"
DST="${DEST_ROOT%/}/date=${RUN_DATE}"
[[ -d "$SRC" ]] || { log "no source dir ${SRC}"; exit 2; }

log "starting ${RUN_DATE}"
mkdir -p "$DST"
if rsync -a "$SRC/" "$DST/"; then
    log "copy finished; verify data before publishing"
else
    status=$?
    log "copy failed with status ${status}"
    exit "$status"
fi

The date check validates the shape only, not calendar validity or the allowed business-date range. Validate those in the job contract. ${VAR:?message} rejects unset or empty input, whereas set -u rejects unset variables. The Bash manual describes the option rules. What the template encodes:

• Stop on error. set -e has exceptions in conditions, &&/|| lists, and some function contexts. Explicitly check critical commands. pipefail exposes upstream failures, but grep returning 1 can simply mean no match; short readers such as head can also cause upstream SIGPIPE.
• Quote variables. "$SRC", not $SRC. Unquoted, a path with a space becomes two arguments, and rm -r $DIR/ with an empty DIR becomes rm -r /. The fourth lab shows the split happening.
• Understand reruns. rsync -a updates selected paths without appending copies, but leaves destination-only files. --delete would remove those files and requires an intentional mirror policy. Neither mode makes a whole dataset update atomic.
• Timestamped logs to stderr, so that stdout can carry data if needed and “when” can be answered later.

Scheduling a script is traditionally done with cron, one line per job in the crontab:

# Example crontab line on Linux with GNU date; use the job's agreed timezone.
# min hour day month weekday   command
15 2 * * * /opt/jobs/load_orders.sh "$(date -d yesterday +\%F)" /landing/orders /raw/orders >> /var/log/load_orders.log 2>&1

In a crontab, an unescaped % has special meaning even inside shell quotes; the example escapes it. GNU date -d is Linux-specific and the business date must use the intended timezone. Cron runs with its own environment and working directory. It schedules times; dependency handling, retries, overlap prevention, and backfills need scripts or other tooling. Mail and system logs can provide basic reporting. See crontab(5).

Part 5: Git

The problem it solves

Without version control, a team shares code by copying folders: etl_v2, etl_v2_final, etl_v2_final_REAL. Nobody knows which one runs in production, two people edit the same file and one set of changes vanishes, and when a number is wrong nobody can say what changed.

Git commits record snapshots of tracked content and links to parent commits, forming a history graph. Untracked files and uncommitted edits are not automatically recoverable snapshots. Branches support parallel work; reviews and deployment records connect that history to what actually ran. Commit authorship and timestamps are useful evidence, not proof of deployment or causality.

The working tree holds editable files. The index (staging area) holds the exact content proposed for the next commit. The commit records that staged snapshot locally; pushing publishes commits to a remote. git diff compares unstaged edits with the index, while git diff --staged compares staged content with HEAD. Editing again after git add leaves that later edit unstaged until you add it again. For example, HEAD can contain version A, the index version B, and the working file version C. An ordinary git commit records B, not C. git diff shows B → C; git diff --staged shows A → B.

The daily loop

git clone git@github.com:org/pipelines.git   # get a copy, once
cd pipelines
git switch -c fix-orders-timezone            # a branch for this piece of work

# ... edit files ...

git status                 # what changed
git diff                   # exactly what changed, line by line
git add sql/orders.sql     # stage what belongs in this commit
git diff --staged          # inspect the snapshot that will be committed
git commit -m "Convert order_ts to UTC before daily aggregate"
git push -u origin fix-orders-timezone
# then open a pull request; a colleague reviews; it is merged into main

Three habits inside that loop do more than any advanced command:

• Read the diff before committing. That is where the debug print, the hardcoded date, and the pasted credential get caught.
• Small commits with messages that say why. “Fix bug” tells the next reader nothing. “Group orders by the agreed UTC day; previous query used the server’s local date” tells whoever investigates in six months what happened, and that person is often you. A 23-hour local day at a daylight-saving transition is not itself an error. UTC is appropriate here because the reporting contract uses UTC; a local business-day report needs the corresponding local boundaries.
• One branch per change, merged through review. In this example team workflow, changes to main go through review. The review is the moment a second person looks at a SQL change before it rewrites a revenue table.

Branches, merging, and conflicts

A branch is a movable reference to a commit; HEAD usually identifies the checked-out branch. main is a team convention, not automatically the deployed version. Git merges compatible changes but can stop on overlapping edits, modify/delete conflicts, or other ambiguities. A normal merge conflict may look like this:

<<<<<<< HEAD
SELECT order_id, amount_local, currency
=======
SELECT order_id, amount_usd
>>>>>>> origin/main
FROM orders;

The block between the first marker and ======= is your version; the block after it is the one you are merging in. Edit the file to the version you want, delete the markers, git add it, and commit. Inspect both changes and test the combined meaning before committing. Use git merge --abort to abandon this merge when appropriate; start merges from a clean working tree. Short-lived branches reduce divergence but do not eliminate conflicts. The sixth lab manufactures one.

First fetch with git fetch origin, then explicitly merge origin/main into the work branch, or rebase unpublished work onto it according to team policy. A bare git pull --rebase uses the branch’s configured upstream, which may be the remote feature branch rather than main. Rebase rewrites commit identities; avoid rewriting shared work without coordination. git-pull documents upstream selection.

Undoing and investigating

git restore file.sql             # restore the working file from the index; discards unstaged edits
git revert abc123                # make a NEW commit that undoes abc123; preserves shared history; may conflict
git log --oneline -20            # recent history
git log -p -- sql/orders.sql     # reachable file history; merge and rename options matter
git blame sql/orders.sql         # who last touched each line, in which commit
git bisect start BAD GOOD        # placeholders: known failing and passing commits
# test the checked-out commit, then git bisect good or git bisect bad
# git bisect reset returns to the original checkout

git log -p and git blame help locate relevant edits. If a revenue figure changed on the 14th and a commit changed a join on the 13th, check the deployed revision, input data, and query behavior before naming that edit as the cause. Blame identifies recorded line history, not personal responsibility for an incident.

With no source or staging option, git restore file.sql restores the working file from the index, so it can discard unstaged edits while retaining staged content; git reset --hard also resets the index and working tree and can discard local work even without moving to an older commit. A force push replaces a remote ref and can overwrite collaborators’ history. On shared history, git revert usually records an undo as a new commit, but it can conflict and does not undo data already changed by a deployed pipeline.

What stays out of Git

.gitignore keeps matching untracked files out of ordinary staging; it does not untrack already committed files or erase history. Explicit staging and diff review still matter. These patterns are examples to adapt: small non-sensitive test fixtures may belong in Git even if bulk CSV data does not. See gitignore.

# secrets
.env
*.pem
credentials.json
# data: Git is for code, not for a 4 GB CSV
*.csv
*.parquet
data/
# local files
__pycache__/
.venv/
.DS_Store

Large or sensitive datasets usually need controlled data storage. Git can compress and share object data, so ten revisions of a 4 GB file do not imply exactly 40 GB, but large binary history can still be costly to clone and maintain. Keep dataset versions or manifests associated with the code; Git LFS and DVC are options for separating tracked references from large objects. They do not remove access-control obligations.

A later deletion does not remove a secret from earlier commits. Treat an exposed credential as compromised: revoke or rotate it first, then assess where it was copied and coordinate cleanup. Rewriting history does not erase existing clones, forks, or caches. GitHub’s sensitive-data guidance explains the cleanup limits. Lab 7 uses a deliberately fake value only.

Lab

Use a disposable workspace with Bash, Python 3, and Git 2.28 or later. Put each lab in its own new directory so files and repositories cannot collide. Labs 1, 3, 4, 6, and 7 are local exercises; labs 2 and 5 require a separate Linux account-administration or SSH environment and are procedural examples. Run code blocks through Bash, not by assuming the current terminal’s shell is Bash. Do not run the catalog of system and remote commands as a batch.

1. Generate a log and interrogate it. Produce 100,000 lines with timestamps, levels (INFO/WARN/ERROR), and varied messages, with an error spike in one hour. Using only pipes: count by level, find the most common error messages, and produce an errors-per-hour table. Time yourself; repeat tomorrow.

export LC_ALL=C
python3 - <<'PY'
import random
from datetime import datetime, timedelta

rng = random.Random(0)
start = datetime(2026, 3, 14)
info = ["job started", "batch loaded", "heartbeat ok", "checkpoint written", "batch committed"]
warn = ["retrying request", "slow query", "disk usage above 90 percent"]
error = ["connection refused to warehouse.internal:5439",
         "timeout reading from vendor api",
         "schema mismatch in orders.csv column 7",
         "out of disk space on /tmp"]
with open("app.log", "w", encoding="utf-8", newline="") as f:
    for i in range(100_000):
        ts = start + timedelta(seconds=i * 0.864)
        weights = [50, 10, 40] if ts.hour == 3 else [90, 8, 2]
        level = rng.choices(["INFO", "WARN", "ERROR"], weights=weights)[0]
        if level == "ERROR":
            message = rng.choices(error, weights=[60, 25, 10, 5])[0]
        elif level == "WARN":
            message = rng.choice(warn)
        else:
            message = rng.choice(info)
        f.write(f"{ts:%Y-%m-%dT%H:%M:%S} {level} worker-{rng.randint(1, 8)} {message}\n")
PY

wc -l app.log
# 100000 app.log
cut -d' ' -f2 app.log | sort | uniq -c
# 3625 ERROR
# 88201 INFO
# 8174 WARN
grep ERROR app.log | cut -d' ' -f4- | sort | uniq -c | sort -rn
# 2179 connection refused to warehouse.internal:5439
#  898 timeout reading from vendor api
#  343 schema mismatch in orders.csv column 7
#  205 out of disk space on /tmp
grep ERROR app.log | cut -c1-13 | sort | uniq -c | sort -rn | head -n 3
# 1680 2026-03-14T03
#  102 2026-03-14T17
#   97 2026-03-14T14

2. Get denied, then fix it. Create a second user. As yourself, create a file with chmod 600. Switch to the second user and try to read it. Fix it with group permissions rather than 777.

# Disposable Linux VM only; the two example names must not already exist.
sudo groupadd de_lab_data
sudo useradd -m -g de_lab_data de_lab_etl
lab_dir="$(mktemp -d /tmp/de-permissions.XXXXXX)"
printf '%s\n' "warehouse.internal" > "$lab_dir/db_host.txt"
chmod 600 "$lab_dir/db_host.txt"
sudo chgrp de_lab_data "$lab_dir" "$lab_dir/db_host.txt"
chmod 750 "$lab_dir"                       # let the target group traverse the directory
sudo -u de_lab_etl cat "$lab_dir/db_host.txt"
# expected: Permission denied (file mode is still 600)

chmod 640 "$lab_dir/db_host.txt"           # owner rw, group r
sudo -u de_lab_etl cat "$lab_dir/db_host.txt"
# warehouse.internal

3. Lie to the scheduler. Write a script that prints an error and exits 0, and a wrapper that runs “next step” only if the first succeeded. Observe the next step run anyway. Fix the exit code.

cat > load.sh <<'EOF'
#!/usr/bin/env bash
echo "ERROR: load failed" >&2
exit 0
EOF
cat > report.sh <<'EOF'
#!/usr/bin/env bash
echo "building report on whatever is in the table"
EOF
chmod +x load.sh report.sh

./load.sh && ./report.sh
# ERROR: load failed
# building report on whatever is in the table
./load.sh; echo "exit code: $?"
# ERROR: load failed
# exit code: 0

sed -i.bak 's/exit 0/exit 1/' load.sh
./load.sh && ./report.sh; echo "exit code of the pair: $?"
# ERROR: load failed
# exit code of the pair: 1

4. Break a script with a space. Create a folder named my data containing two .tmp files. Write a script that lists $DIR/*.tmp without quotes and run it with DIR="my data". Read what it would have deleted. Add quotes and set -u, then run it with DIR unset.

mkdir -p "my data" && touch "my data/a.tmp" "my data/b.tmp"

cat > cleanup.sh <<'EOF'
#!/usr/bin/env bash
printf 'argument: %s\n' $DIR/*.tmp
EOF
cat > cleanup_fixed.sh <<'EOF'
#!/usr/bin/env bash
set -u
printf 'argument: %s\n' "$DIR"/*.tmp
EOF
chmod +x cleanup.sh cleanup_fixed.sh

DIR="my data" ./cleanup.sh
# argument: my
# argument: data/*.tmp
DIR="my data" ./cleanup_fixed.sh
# argument: my data/a.tmp
# argument: my data/b.tmp
env -u DIR ./cleanup_fixed.sh; echo "exit code: $?"
# ./cleanup_fixed.sh: line 3: DIR: unbound variable
# exit code: 1

5. Survive a disconnect. Over SSH, start a five-minute job inside tmux, close the terminal, reconnect, and reattach.

ssh user@host
tmux new -s etl                                       # a named session
for i in $(seq 1 300); do echo "tick $i"; sleep 1; done | tee ticks.log
# close the terminal window while it is running, then from a new terminal:
ssh user@host
tmux ls                                               # etl: 1 windows ...
tmux attach -t etl                                    # the loop is still counting
# Ctrl-b d detaches and leaves it running; exit inside the session ends it

6. Cause and resolve a conflict. Clone a repository into two folders, change the same line in each, commit both, and merge. Resolve by hand. Then use git log -p and git blame to reconstruct what happened.

git init -q --bare -b main hub.git
# All clones and pushes below are local, inside the scratch directory.
git clone -q hub.git alice
cd alice
git config user.name Alice
git config user.email alice@example.com
printf 'SELECT order_id, amount\nFROM orders;\n' > orders.sql
git add orders.sql
git commit -qm "Add orders query"
git push -q -u origin main
cd ..
git clone -q hub.git bob                  # Bob receives the original commit

cd alice
sed -i.bak 's/amount/amount_usd/' orders.sql
rm orders.sql.bak
git commit -qam "Report amounts in USD"
git push -q
cd ../bob
git config user.name Bob
git config user.email bob@example.com
sed -i.bak 's/amount/amount_local, currency/' orders.sql
rm orders.sql.bak
git commit -qam "Keep local currency alongside amount"
git fetch -q origin
git merge origin/main
# Auto-merging orders.sql
# CONFLICT (content): Merge conflict in orders.sql
# Automatic merge failed; fix conflicts and then commit the result.
cat orders.sql
# <<<<<<< HEAD
# SELECT order_id, amount_local, currency
# =======
# SELECT order_id, amount_usd
# >>>>>>> origin/main
# FROM orders;

printf 'SELECT order_id, amount_usd, amount_local, currency\nFROM orders;\n' > orders.sql
git add orders.sql
git commit -qm "Merge: keep both USD and local amounts"
git log --topo-order --format='%s'
# Merge: keep both USD and local amounts
# Report amounts in USD
# Keep local currency alongside amount
# Add orders query
git diff HEAD^1 HEAD -- orders.sql | grep -E '^[-+]SELECT'
# -SELECT order_id, amount_local, currency
# +SELECT order_id, amount_usd, amount_local, currency
git blame --line-porcelain orders.sql | grep '^summary '
# summary Merge: keep both USD and local amounts
# summary Add orders query

7. Leak a secret, then recover. In a throwaway repository, commit a fake API key, delete it in a second commit, and show with git log -p that it is still there. Write down the two steps you would take in real life, in order.

git init -q -b main leak
cd leak
git config user.name Dev
git config user.email dev@example.com
printf 'API_KEY=FAKE_DEMO_NOT_A_REAL_CREDENTIAL\n' > config.env
git add config.env
git commit -qm "Add vendor config"
git rm -q config.env
git commit -qm "Remove committed secret"
git status --short
# no output: the working tree and index are clean
git log -p --format='--- %s' | grep -E '^[-+]API_KEY='
# -API_KEY=FAKE_DEMO_NOT_A_REAL_CREDENTIAL
# +API_KEY=FAKE_DEMO_NOT_A_REAL_CREDENTIAL