Frozen Encoder

Definition

A frozen encoder is a pretrained encoder model, such as BERT, ResNet, or a sentence transformer, whose weights are not updated during training on a downstream task.

• The encoder is frozen, meaning its parameters remain fixed.
• Only the trainable components added on top, such as a classification or regression head, are updated.
• The encoder acts as a fixed feature extractor, transforming inputs into representations used by the downstream model.

---

Why Freeze an Encoder?

1. Efficiency
   • Fewer parameters require gradient updates, which can make training faster.
   • Less memory is generally needed for storing gradients and optimizer states.
2. Small Data Settings
   • Freezing can reduce the risk of overfitting when labeled data is limited.
   • It preserves representations learned from a large pretraining dataset.
3. Transfer Learning Strategy
   • First stage: freeze the encoder and train only the task-specific head.
   • Later stage: unfreeze some or all encoder layers and fine-tune them using a smaller learning rate.

---

Examples

1. NLP (BERT)
   • Use a pretrained BERT encoder to convert text into contextual representations.
   • Keep the BERT weights frozen.
   • Train a classification head on the representations for sentiment classification.
2. Computer Vision (ResNet)
   • Use an ImageNet-pretrained ResNet50 as a feature-extraction backbone.
   • Freeze the pretrained backbone.
   • Train a new classification head for a medical image dataset.
3. Recommendation Systems
   • Use a pretrained encoder to generate fixed user or item representations.
   • Train only the downstream scoring or ranking model on task-specific data.

---

Frozen vs. Fine-Tuned

• Frozen encoder: The encoder weights remain fixed while the task-specific components are trained.
• Partially fine-tuned encoder: Only selected encoder layers, often the final layers, are updated.
• Fully fine-tuned encoder: All encoder weights are updated using the downstream dataset.

Fine-tuning can improve task-specific performance, especially when the downstream data differs from the pretraining data. However, it generally requires more computation and may increase the risk of overfitting on small datasets.

---

Analogy

Think of a frozen encoder as a feature extractor:

• It is similar to using fixed, pretrained Word2Vec or GloVe embeddings without updating them.
• A smaller task-specific model is trained using the features produced by the encoder.

---

Summary
A frozen encoder is a pretrained encoder whose weights remain fixed during downstream training. It serves as a feature extractor, reduces computational requirements, and can help limit overfitting when labeled data is scarce. Some or all of its layers can later be unfrozen for task-specific fine-tuning.