Categorical Explosions

A categorical explosion occurs when a feature with many distinct values expands into a very wide representation. With one-hot encoding, each category has its own indicator column. A product field with 50,000 known identifiers can therefore require 50,000 columns, even though each row contains only one product. The difficulty combines storage cost, limited evidence for rare categories, and handling categories that arrive after training.

High cardinality means many distinct values. It does not by itself make a feature unusable. A frequently purchased product may have a useful history, while an order identifier that appears only once may offer no repeatable signal. Start by asking what information the identifier carries and which entities will recur when the model is used.

Why one-hot encoding grows

Suppose the vocabulary is red, blue and green. One-hot encoding represents blue as [0, 1, 0]. A linear model can learn a different contribution for each column without pretending that blue lies numerically between red and green. Replacing the categories with integers 1, 2 and 3 saves columns, but a model that treats those numbers as quantities introduces an order and distances that the categories do not have.

For 200,000 rows and 50,000 categories, a dense matrix contains ten billion entries. At eight bytes per entry, that is 80 billion bytes: 80 GB, or about 74.5 GiB. Almost all entries are zero. A sparse representation stores nonzero values and their locations instead of allocating every zero.

For a CSR matrix with exactly one nonzero per row, eight-byte values, four-byte column indices and four-byte row pointers, the main arrays occupy 200,000 × (8 + 4) + 200,001 × 4 = 3,200,004 bytes, about 3.05 MiB. This illustrative calculation excludes object overhead and vocabulary storage. Different numeric types, index widths and intermediate transformations change the total. Check whether the chosen estimator and every preprocessing step accept sparse input; an intermediate dense conversion can recreate the large allocation.

Storage does not create statistical evidence

A sparse matrix solves the representation problem. It does not give a rare category more observations. In an unregularized linear model, a category-specific coefficient supported by only two rows is highly sensitive to those rows. Regularization can shrink that coefficient toward a baseline, but the estimate then depends heavily on the shrinkage assumption. Other models allocate parameters differently, yet still need evidence to distinguish a reliable pattern from an accident.

Separate categories that are rare from categories that are unseen. A rare category has some training evidence; an unseen category has none. An encoder can map an unseen value to an unknown bucket or an all-zero vector, but this is a fallback policy, not a learned estimate of that new category’s effect. Evaluate with the kind of novelty expected in deployment. A random row split can look reassuring when the same users occur in both sets, while a new-user application needs evaluation on held-out users.

Pooling categories and compressing their representation

Domain grouping pools values that share meaningful structure: postcodes into regions or products into product families. This lets related categories share evidence, at the cost of losing differences within a group. The grouping is an assumption to test. Two neighboring areas can have different behavior, and a broad product family can conceal a useful distinction.

Rare-category pooling instead puts low-frequency values into an “other” group. It preserves separate effects for common categories and stabilizes the tail by pooling observations. Choose the cutoff using training data and validation performance. In scikit-learn, OneHotEncoder supports min_frequency and max_categories; routing unseen values into an infrequent bucket also depends on handle_unknown and whether such a bucket exists. Save the fitted vocabulary with the model so training and prediction use the same mapping.

Feature hashing maps each category name deterministically into a fixed number of columns. It avoids maintaining a growing vocabulary and can process new names immediately. Collisions are the trade-off: if two categories map to the same bucket, their effects cannot be represented independently in that column. Signed hashing can reduce some aggregate collision bias, but cannot recover an individual distinction lost to a collision. Wider tables reduce collision pressure while increasing the model’s footprint.

Even more buckets than categories do not guarantee unique assignments. Distinguish the number of excess assignments from the number of categories affected: if three categories share one bucket, there are two assignments beyond the first, but all three categories are involved. A collision rate is uninterpretable unless its numerator is stated. Validate predictive performance as well as memory; a collision involving an important category can matter much more than several collisions in an irrelevant tail.

What other encodings assume

Frequency encoding replaces a category with its observed count or frequency. It expresses how common the category is, and loses identity when categories have the same frequency. Fit counts on the permitted training history. Target encoding instead estimates a category’s average outcome, usually shrunk toward an overall mean. It can be compact and informative, but a training row must not receive an encoding computed using its own target. Out-of-fold or appropriately ordered construction addresses that leakage; validation and future rows use mappings fitted only from the allowed training data.

An embedding learns a vector for each category, allowing a downstream model to use several learned properties instead of a single indicator. Its parameter table still grows with vocabulary size, and rare categories may have poorly estimated vectors. Side information or shared structure can help with new entities; merely assigning a new embedding index does not solve cold start. Models with native categorical support offer another route, but their category handling and constraints differ. Confirm the estimator’s documented behavior rather than assuming that accepting integer IDs makes them categorical.

Before encoding a high-cardinality field, distinguish nominal identity from meaningful order or structure. Unique timestamps can still yield trend and seasonal features. A location or time proxy becomes leakage when its construction or availability crosses the evaluation boundary, not simply because it has many values. The useful comparison is whether the representation retains relevant information, has enough evidence behind it, and behaves sensibly for future categories.

References

• scikit-learn API reference, FeatureHasher
• scikit-learn API reference, OneHotEncoder
• SciPy reference, Sparse arrays