Embedding
Definition
An embedding is a way of representing data (words, images, users, products, etc.) as a vector of numbers in a continuous space.
- Each vector captures the meaning or characteristics of the object.
- The goal is that similar objects → similar vectors (close in space).
Why Use Embeddings?
- Raw data (text, images) is high-dimensional and hard for models to use directly.
- Embeddings turn them into dense, low-dimensional representations that machine learning models can process efficiently.
Examples
- Word Embeddings (NLP)
- Words like “king” and “queen” are close in vector space.
- Famous relationship:
- $\text{vector}(\text{king}) – \text{vector}(\text{man}) + \text{vector}(\text{woman}) \approx \text{vector}(\text{queen})$
- Image Embeddings (CV)
- A CNN maps an image into a vector.
- Images of cats cluster near each other; dogs cluster elsewhere.
- Recommendation Systems
- User embedding: summarizes a user’s behavior.
- Item embedding: summarizes product characteristics.
- Recommendation = nearest neighbor search in embedding space.
How They’re Learned
- Supervised: Through classification (e.g., BERT fine-tuned for sentiment).
- Unsupervised: Word2Vec, GloVe, autoencoders learn structure without labels.
- Contrastive learning: Push similar pairs together, dissimilar pairs apart (e.g., SimCLR, CLIP).
Properties
- Dense vectors (as opposed to sparse one-hot encoding).
- Typically 50–1000 dimensions.
- Preserve semantic similarity.
Applications
- Natural Language Processing (NLP) → word, sentence, document embeddings.
- Computer Vision (CV) → face recognition, image search.
- Recommender Systems → matching users with items.
- Clustering & Visualization → embeddings reveal structure in data.
- Transfer Learning → pretrained embeddings used in new tasks.
Example Intuition
- Imagine mapping words into a 2D plane:
- Words related to animals cluster in one area.
- Words related to food cluster in another.
- The embedding captures these relationships automatically.
Summary
Embeddings are vector representations of complex data that capture similarity and meaning. They’re the backbone of modern ML for text, images, and recommendations, enabling models to generalize and transfer knowledge efficiently.