Definition

An autoencoder is a type of neural network designed to learn a compressed representation (encoding) of input data and then reconstruct it back as output.

  • Goal: capture the most important features of the input in a lower-dimensional form.
  • Commonly used for dimensionality reduction, denoising, anomaly detection, and representation learning.

Architecture

  1. Encoder
    • Maps input $x$ into a lower-dimensional vector $z$ (the latent representation).
    • Example: $z = f_{\theta}(x)$.
  2. Latent Space (Code / Bottleneck)
    • The compressed representation of the input.
    • Dimension is usually much smaller than input.
  3. Decoder
    • Reconstructs the original input from $z$.
    • Example: $\hat{x} = g_{\phi}(z)$.
  4. Loss Function
    • Measures how close the reconstruction $\hat{x}$ is to the original $x$.
    • Typically Mean Squared Error (MSE) or Cross-Entropy Loss.

Training Objective

$\min_{\theta, \phi} \; L(x, \hat{x}) = \min_{\theta, \phi} \; L \big( x, g_{\phi}(f_{\theta}(x)) \big)$

  • Encoder ($\theta$) and Decoder ($\phi$) are trained together to minimize reconstruction error.

Variants of Autoencoders

  1. Vanilla Autoencoder → basic structure (encoder + decoder).
  2. Denoising Autoencoder → trained to reconstruct clean data from noisy input.
  3. Sparse Autoencoder → enforces sparsity in latent space (most activations = 0).
  4. Variational Autoencoder (VAE) → probabilistic latent space, good for generative modeling.
  5. Convolutional Autoencoder → uses CNNs, often applied to images.

Applications

  • Dimensionality Reduction → alternative to PCA.
  • Denoising → removing noise from images/text/speech.
  • Anomaly Detection → if reconstruction error is high, input may be anomalous (fraud, network intrusion, medical anomaly).
  • Data Compression → reducing storage needs.
  • Representation Learning → learn embeddings useful for downstream ML tasks.
  • Generative Modeling → VAEs generate new samples.

Example

  • Input: Handwritten digit image (28×28 pixels).
  • Encoder compresses into a 32-dimensional latent vector.
  • Decoder reconstructs the image back.
  • If trained well, the output image looks very similar to the original, but the latent code captures the essence of the digit.

Summary
An autoencoder is a neural network that learns to compress data into a latent representation and then reconstruct it. It’s useful for feature extraction, anomaly detection, denoising, and generative modeling (especially with VAEs).