Metric Cardinality

Metric cardinality is the number of distinct label combinations a metric produces. Each exported series consumes collection, storage, and query resources. Counting metric names alone therefore understates the load when a name has many combinations.

The word is overloaded, so one clarification first. In statistics and feature engineering, cardinality describes how many distinct values a categorical variable takes; in data modeling, join cardinality describes the multiplicity between two tables. This entry is about neither. It is about the multiplication that happens when you attach dimensions to a time series.

It multiplies, which is the whole problem

Prometheus recommends labels for dimensions that need to be aggregated or filtered together. A response counter with a status-code label is easier to query as a family than independently named counters for each code. Avoid generating a new metric name for each customer or request; moving the identifier into a label still requires checking its cardinality.

The cost is that dimensions compose by multiplication. A request counter labeled by five status codes, four HTTP methods and twenty endpoints is not twenty-nine series; it is up to 400, times however many instances export it. Adding one more dimension with ten values makes it 4,000. Nothing warns you, because each individual label looked reasonable.

The documentation’s filesystem example has roughly ten series per node, or 100,000 across 10,000 nodes. Adding per-user quotas for 10,000 users can create tens of millions of series. The exact populated combinations matter, but the product illustrates why one extra dimension can dominate the total. Its warning about Prometheus capacity describes the documentation’s implementation context, not a benchmark for every installation.

Even before an absolute limit is reached, these series compete with other telemetry for memory, CPU, disk, network bandwidth, and query capacity. Measure the cost against the questions the dimension enables.

Metric types add their own storage structure. A classic histogram exports bucket series plus sum and count; twelve bucket series, including the positive-infinity bucket, therefore give fourteen series per label combination. If twelve finite boundaries are configured, include the additional infinity bucket when counting. A summary exports its configured quantiles plus sum and count. Its precomputed quantiles cannot be combined to obtain a fleet quantile, but its sum and count can be aggregated for counts and averages. A native histogram uses a composite sample in one series, whose size still depends on its buckets; one series does not mean constant byte cost.

How much is safe

Prometheus’s conservative instrumentation guidance suggests keeping most metrics below ten label combinations and limiting exceptions. Use this as a prompt to examine each dimension’s purpose and growth, not as a universal safe capacity for an entire backend.

For a metric that can exceed a hundred combinations, the documentation recommends investigating alternatives such as reducing dimensions or moving detailed analysis elsewhere. It does not require every such metric to be removed. Assess combinations across instances, churn as values change, retention, scrape frequency, query workload, and available resources.

Which points at the real criterion, and it is not importance. Ask whether the set of values is bounded by something other than your traffic. HTTP status codes are bounded by a specification. Deployment environments are bounded by a list someone maintains. Customer identifiers, request identifiers, session identifiers, full URL paths and error message strings are bounded only by how successful you are, which means their cost grows with the thing you want to grow.

The SRE workbook describes an App Engine team that exported the bounded HTTP status-code vocabulary as a label, making the signal available for graphs and alerts. A different team needed entity IDs with millions of possible values and kept that detail in logs instead. The cases illustrate evaluating both usefulness and expected series growth.

The second team documented a log-query script for responders. Logs or traces often suit individual identifiers better than metric labels, but they still incur ingestion, indexing, retention, and access-control costs. Choose the signal from the analysis required and its resource budget rather than assuming unbounded values are cheap elsewhere.

Keeping it under control

Start with dimensions needed by concrete queries, alerts, or investigations and add others deliberately. Removing a dimension can affect existing consumers, so record those dependencies when adding it.

Keep that review part of instrumentation and operations:

• Identify a consumer. Name the question a metric and its dimensions answer, including infrequent diagnostic use. A field without a current metric use may belong in retained event detail, subject to its own cost and access constraints.
• Bound the values at the instrumentation boundary. Map a full URL path onto its route template, group status codes into classes where the class is what you alert on, and replace an identifier with a category. A label whose values you can enumerate on request is one you control; one that is passed through from user input is not.
• Choose resolution and retention together. Collect fine observations when brief changes matter, then use an aggregation that preserves the relevant distribution or extrema. A minute average alone does not preserve every short hotspot.
• Review removal candidates. Retire unused dimensions or signals after checking their alerting and diagnostic dependencies. Reduction, aggregation, and retention changes offer alternatives when the information still matters.

When the backend struggles, examine series counts and growth by metric and label, along with churn, ingestion, queries, and resource saturation. An unexpected identifier label is one possible cause; the data should establish whether a small number of metrics or broad growth is responsible.

How this fits the wider decision about what to measure, what to page on and what to keep for diagnosis is worked through in Only a Few Times a Day; the three kinds of telemetry and what each is good at are summarized in telemetry signals.

References

• Prometheus — Instrumentation
• Prometheus — Histograms and summaries
• Google SRE Workbook — Monitoring
• “Monitoring Distributed Systems,” Google SRE Book

The numeric guidelines and the stated implementation limit are product documentation and can change.