DataOps
DataOps is the practice of applying software engineering and operations discipline to analytics work: keeping transformation logic in version control, testing it automatically, deploying it through a repeatable process, and monitoring the results in production. The DataOps Manifesto, published and stewarded by DataKitchen, states it as a set of principles rather than a specification, and it is worth reading as a position argued from experience rather than as a measured result.
What the manifesto asserts
- Analytics is code. The tools an analytic team uses to access, integrate, model, and visualize data all produce code and configuration describing what is done to the data — so that output can be managed like code.
- Version everything. “Reproducible results are required and therefore we version everything: data, low-level hardware and software configurations, and the code and configuration specific to each tool in the toolchain.”
- Monitor quality continuously, with performance, security, and quality measures watched to detect unexpected variation and produce operational statistics.
- Orchestrate end to end. The manifesto names the orchestration of data, tools, code, environments, and the team’s work as a key driver of analytic success.
- Shorten the cycle, minimizing the time from a stated need to a repeatable production process.
Why it is not just DevOps with data in the name
| Difference | Consequence |
|---|---|
| A deployment changes the past | New transformation logic recomputes history, so shipping can move numbers people already reported |
| The input is not controlled | Code can be correct and the output still wrong because an upstream system changed what it sends |
| Correctness is not observable from the process | A green run says the code executed; whether the data is right needs separate assertions |
| Test data is hard to fake | The interesting bugs live in real data’s edge cases, which pushes testing toward production-like data and raises access questions |
Versioning data itself, in the literal sense the manifesto asks for, is the expensive half. Most teams approximate it: immutable raw storage, rebuildable partitions, snapshots of the dimensions that change, and a record of which code version produced which table.
What no tool installs
The technical practices are adoptable in an afternoon and often are — a repository, a CI job, a test suite. What stays unresolved is the allocation of responsibility: who owns a given table, who is called when its build fails, who decides that a metric’s definition changes, and how consumers are told when it does. A team can hold every practice in the manifesto and still ship a redefinition of revenue that nobody with authority over revenue approved.
Note also that the term is used both for this set of practices and as a product category by vendors selling tools; those are not the same claim. How the practices and the ownership questions fit together is worked through in Transformation as Code.
References: DataKitchen, The DataOps Manifesto.
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.