Quality Threshold
A threshold needs a defined population, window, and acceptable impact. Derive its value from the decision it protects rather than choosing a percentage without a basis.
-
-
Data Quality and Data Contracts
A green test suite means the data broke none of the rules you thought to write. Whether the numbers are correct is a different question with a different owner.
-
Fault Injection
Fault injection exercises recovery paths before an incident. A refund example shows why an apparently successful reply can hide a repeated side effect.
-
Designing Maintainable Data Workflows
How to structure a data job so its inputs are visible, its effects are local, and changes can be checked: dependency injection, contract tests, configuration precedence and validation, and effective-configuration records.
-
Testing and Debugging Spark Data Pipelines
Build fixtures and independent expected results, test real storage boundaries, and reduce Spark failures into useful regression tests.
-
Edge Case
Learn what an edge case is, why boundary rows break rules that work on typical data, and how to decide the behavior instead of discovering it.
-
Software Engineering Habits for Data Work
Make data changes easier to verify and recover with tests, focused reviews, incremental refactoring, and useful operational documents.