DataOps
DataOps is the practice of applying software engineering and operations discipline to analytics work: keeping transformation logic in version control, testing it automatically, deploying it through a repeatable process, and monitoring the results in production. The DataOps Manifesto, published and stewarded by DataKitchen, states it as a set of principles rather than a specification.
What the manifesto asserts
- Analytics is code. The tools an analytic team uses to access, integrate, model, and visualize data all produce code and configuration describing what is done to the data — so that output can be managed like code.
- Version everything. “Reproducible results are required and therefore we version everything: data, low-level hardware and software configurations, and the code and configuration specific to each tool in the toolchain.”
- Monitor quality continuously, with performance, security, and quality measures watched to detect unexpected variation and produce operational statistics.
- Orchestrate end to end. The manifesto names the orchestration of data, tools, code, environments, and the team’s work as a key driver of analytic success.
- Shorten the cycle, minimizing the time from a stated need to a repeatable production process.
Why it is not just DevOps with data in the name
| Difference | Consequence |
|---|---|
| A deployment can change the past | When a rebuild, backfill, or re-evaluation applies new transformation logic to earlier periods, numbers people already reported can move. Whether history is reprocessed is a decision separate from the deployment |
| The input is not controlled | Code can be correct and the output still wrong because an upstream system changed what it sends |
| Correctness is not observable from the process | A green run says the code executed; whether the data is right needs separate assertions |
| Test data is hard to fake | The interesting bugs live in real data’s edge cases, which pushes testing toward production-like data and raises access questions |
Versioning data itself, in the literal sense the manifesto asks for, is the expensive half. Most teams approximate it: immutable raw storage, rebuildable partitions, snapshots of the dimensions that change, and a record of which code version produced which table.
What no tool installs
The technical practices are the adoptable part — a repository, a CI job, a test suite, monitoring — though setting them up well is work rather than an afternoon. What stays unresolved is the allocation of responsibility: who owns a given table, who is called when its build fails, who decides that a metric’s definition changes, and how consumers are told when it does. A team can hold every practice in the manifesto and still ship a redefinition of revenue that nobody with authority over revenue approved.
Note also that the term is used both for this set of practices and as a product category by vendors selling tools; those are not the same claim. How the practices and the ownership questions fit together is worked through in Transformation as Code.
References
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.