Standard Streams and Redirection

Standard streams are the conventional input and output connections of a command. On Unix-like systems, file descriptor 0 is standard input, 1 is standard output, and 2 is standard error. Commands normally inherit these connections, but they can close them or redirect them to terminals, files, pipes, or other resources.

Standard output conventionally carries results that another tool can consume. Standard error carries diagnostics, which can include progress rather than failure. An ordinary shell pipe connects one command’s stdout to the next command’s stdin; it does not automatically include stderr. Mixing diagnostics into a data stream can break the downstream parser.

Redirection changes those connections. The operator > writes stdout to a file and normally truncates an existing file; >> appends. The operator 2> redirects stderr. Redirections are evaluated from left to right. In > all.txt 2>&1, stderr is made to use stdout’s already redirected destination. Reversing the order can leave stderr at the original stdout destination while only stdout goes to the file.

The shell opens the output destination before running the command. Reading a file while redirecting output onto that same file can therefore erase the input before processing begins. Use a separate output and verify it before replacement. A pipe transports bytes; newline-separated records are a convention used by many text tools, not a property that makes arbitrary CSV or binary data safe to process line by line.

Reference: GNU Bash: Redirections. Practice these ideas in Linux, the Shell, and Git for Data Work.