Descriptive and Diagnostic Analytics

Descriptive analytics answers what happened: revenue by month, active users by segment, order completion rate this week against last. Diagnostic analytics answers why it happened. The two are usually mentioned together, and the step between them is the largest one in analytical work.

The reason is that they need different things. Describing needs agreed definitions and data you trust. Explaining needs a causal claim, and no volume of description produces one.

Description is harder than it sounds, for a boring reason

The difficulty is rarely computation. It is that “active user” means one thing to the growth team and another to finance, “revenue” may or may not be net of refunds, and last month’s number changed because a late-arriving batch landed after the report ran.

Which is why most of the value in this category comes from metric definition and data reliability rather than from technique, and why a team whose descriptive layer is disputed cannot build anything credible on top of it. Two people arguing about whether a number is right are not going to agree about why it moved.

Good descriptive work also states its own limits: the definition used, the period, what is excluded, and when the figure was computed. A number without those is not wrong so much as unusable by anyone who was not in the room.

Slicing generates candidates, not causes

The standard diagnostic move is to break a change down — by region, channel, segment, device, cohort — until something stands out. It is a good first step and it cannot finish the job, because everything that changed in the same period shows up in the slices whether or not it caused anything.

Three explanations fit the same slice equally well, and the data cannot separate them.

• The slice is the cause. The channel’s performance dropped and that dragged the total down.
• The slice is a consequence. Demand fell for another reason, and this channel simply registered it first.
• Both are symptoms. A third factor — a pricing change, a competitor, a seasonal shift — moved both.

There is a further trap specific to slicing: a relationship present in the whole can reverse within subgroups, and vice versa, depending on how the groups are sized. So the result you find depends partly on which breakdown you happened to try, and the practice of slicing until something looks responsible is a search procedure that will always return an answer. Correlation within a slice is evidence about association and nothing more.

Say that the question is causal

The most useful habit is naming the question honestly. In The C-Word, Miguel Hernán uses wine and heart disease to distinguish an observed association from the effect of changing drinking behavior. If the aim is to estimate that effect, describing it only as an association hides the question the analysis must answer. His argument applies to observational studies as well as randomized experiments: making the causal aim explicit helps align the design, analysis and interpretation.

Once the question is named, three routes exist and they differ in strength rather than in kind: an experiment, where the change can be randomized; a causal inference analysis of existing data, at the price of assumptions that must be stated; or a reasoned argument from timing, mechanism, and eliminated alternatives. The third is what most organizations actually do, and the improvement available is not a fancier method but writing the reasoning down — including what would have to be true for it to be wrong.

The output formats differ accordingly. A descriptive answer is a number with its definition. A diagnostic answer is a claim, the evidence for it, the alternatives considered and why they were rejected, and a statement of how confident it is. Delivering the second in the format of the first — a chart with a title asserting the cause — is the most common way analysis misleads people who have no way to tell.

How these two fit alongside predictive and prescriptive questions, and which capability each needs, is worked through in What Kind of Question Is It?.

• Hernán, The C-Word: Scientific Euphemisms Do Not Improve Causal Inference From Observational Data, American Journal of Public Health 108(5), 2018.