Definition

Representation shift happens when the internal representation of data (e.g., learned embeddings, feature vectors) changes over time between training and deployment, even if the raw input distribution looks similar.

It’s a special case of distribution shift but focuses on feature/embedding space instead of raw input.

Where It Appears

  1. Neural Networks / Embeddings
    • Models learn hidden representations (e.g., word embeddings, image embeddings).
    • If the mapping changes (due to retraining, fine-tuning, or new data), downstream tasks may fail.
  2. Preprocessing Pipelines
    • A feature engineering step (e.g., TF-IDF, PCA, scaling) may produce different distributions over time.
    • Example: TF-IDF weights drift because new vocabulary dominates incoming text.
  3. Domain Shift in Inputs
    • Even if inputs look “similar,” their embedding distributions can drift.
    • Example: A face recognition model trained on frontal faces, but deployed in an environment with many side-profile faces → the embedding space shifts.

Impact on Models

  • Downstream classifiers/regressors (that assume stable representations) degrade.
  • Similarity search (e.g., nearest-neighbor in embedding space) gives wrong results.
  • Fairness issues if certain groups’ embeddings shift differently over time.

Detection Methods

  1. Distance Metrics on Embeddings
    • Compare representation distributions using MMD, Energy Distance, KL divergence, etc.
    • Track cosine similarity or Euclidean distances between historical and new embeddings.
  2. Visualization
    • Use t-SNE / UMAP to see if clusters shift across time.
  3. C2ST (Classifier Two-Sample Test)
    • Train a classifier to distinguish old embeddings vs. new embeddings.

Example

  • In NLP: Word embeddings for “mask” in 2019 mostly mean “cosmetic mask,” but in 2020 (COVID) it shifts toward “face mask.”
    → The representation shift affects downstream sentiment or topic models.
  • In recommender systems: A product embedding space may shift as new product categories appear, causing similarity recommendations to break.

Summary
Representation shift is when the feature or embedding space distribution changes, not just raw data. It breaks downstream models that rely on stable representations, so monitoring embedding drift is as important as monitoring input drift.