Unit and Integration Tests

A unit test checks a small unit of behavior in controlled conditions. An integration test checks how components work together across a boundary, such as a parser and file format library or an application and database. Teams draw the boundary differently; state what is real, what is replaced, and what behavior the test exercises.

Pure functions make focused tests easier because the same inputs produce the same result without observable side effects. Test a per-currency total with a few explicit orders and expected amounts, rather than calculating the expected answer with the same transformation being tested. Include a nearby counterexample such as an order just outside the intended UTC day.

A fixture supplies a controlled starting state. A temporary directory lets a file test create output without depending on production data. Reading Parquet back can check values and schema, including empty output. Repeating the write can test the specified rerun behavior; simulating a failed write can check that an existing output remains intact.

A local substitute has limits. SQLite can exercise a query pattern but cannot prove a warehouse’s dialect, authentication, or transaction behavior. Add checks against the actual integration for those contracts. Passing tests establish the exercised cases, not every possible input. Linting, formatting, and static typing answer other questions and complement rather than replace behavioral tests.

Reference: pytest fixtures. For worked examples, see Python for Data Engineers.