Autoencoder

Definition

An autoencoder is a type of neural network designed to learn a compressed representation (encoding) of input data and then reconstruct it back as output.

• Goal: capture the most important features of the input in a lower-dimensional form.
• Commonly used for dimensionality reduction, denoising, anomaly detection, and representation learning.

---

Architecture

1. Encoder
   • Maps input $x$ into a lower-dimensional vector $z$ (the latent representation).
   • Example: $z = f_{\theta}(x)$.
2. Latent Space (Code / Bottleneck)
   • The compressed representation of the input.
   • Dimension is usually much smaller than input.
3. Decoder
   • Reconstructs the original input from $z$.
   • Example: $\hat{x} = g_{\phi}(z)$.
4. Loss Function
   • Measures how close the reconstruction $\hat{x}$ is to the original $x$.
   • Typically Mean Squared Error (MSE) or Cross-Entropy Loss.

---

Training Objective

$\min_{\theta, \phi} \; L(x, \hat{x}) = \min_{\theta, \phi} \; L \big( x, g_{\phi}(f_{\theta}(x)) \big)$

• Encoder ($\theta$) and Decoder ($\phi$) are trained together to minimize reconstruction error.

---

Variants of Autoencoders

1. Vanilla Autoencoder → basic structure (encoder + decoder).
2. Denoising Autoencoder → trained to reconstruct clean data from noisy input.
3. Sparse Autoencoder → enforces sparsity in latent space (most activations = 0).
4. Variational Autoencoder (VAE) → probabilistic latent space, good for generative modeling.
5. Convolutional Autoencoder → uses CNNs, often applied to images.

---

Applications

• Dimensionality Reduction → alternative to PCA.
• Denoising → removing noise from images/text/speech.
• Anomaly Detection → if reconstruction error is high, input may be anomalous (fraud, network intrusion, medical anomaly).
• Data Compression → reducing storage needs.
• Representation Learning → learn embeddings useful for downstream ML tasks.
• Generative Modeling → VAEs generate new samples.

---

Example

• Input: Handwritten digit image (28×28 pixels).
• Encoder compresses into a 32-dimensional latent vector.
• Decoder reconstructs the image back.
• If trained well, the output image looks very similar to the original, but the latent code captures the essence of the digit.

---

Summary
An autoencoder is a neural network that learns to compress data into a latent representation and then reconstruct it. It’s useful for feature extraction, anomaly detection, denoising, and generative modeling (especially with VAEs).