Autoencoders and Representation Learning

An autoencoder learns to reconstruct its input through an intermediate code. A smaller code, a sparsity penalty, or a corrupted input changes what information the model can pass to the decoder. Those choices determine which patterns it preserves. A low reconstruction error alone does not tell us whether the code will help a later task.

Encoder, code, decoder

An encoder \(f\) maps input \(x\) to a code \(z=f(x)\), and a decoder \(g\) produces a reconstruction \(\hat x=g(z)\). An undercomplete code has fewer coordinates than x; other variants can use as many or more. Basic training minimizes reconstruction error without externally supplied labels: each input supplies its own target. Both networks receive gradients through the same reconstruction loss.

For representation learning, we keep the encoder and use its code as features for another task. For denoising or restoration, the reconstructed output is itself useful. The code below accepts rows of flattened features: eight 28-by-28 grayscale images would have shape (8,784), and the encoder produces eight 32-coordinate codes. Random inputs here check those shapes; they do not train the model. Run the Python blocks in order.

import torch
import torch.nn as nn

class Autoencoder(nn.Module):
    def __init__(self, in_dim=784, hidden=256, code_dim=32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(in_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, code_dim))
        self.decoder = nn.Sequential(
            nn.Linear(code_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, in_dim))

    def forward(self, x):
        z = self.encoder(x)
        return self.decoder(z), z

model = Autoencoder()
x = torch.randn(8, 784)
recon, z = model(x)
print(recon.shape, z.shape)
# torch.Size([8, 784]) torch.Size([8, 32])

The output layer must be able to represent the target range. A linear output with MSE can reconstruct real-valued or standardized features. Bounded targets can use a sigmoid output with MSE; binary targets can use a Bernoulli loss, implemented stably as BCEWithLogitsLoss on raw logits. Scaling continuous pixels to [0,1] does not by itself make them Bernoulli observations, though binary cross-entropy is also used as a surrogate. A sigmoid cannot reconstruct negative standardized values. In the examples below, MSE averages over examples and coordinates: reconstructing [1,3] as [2,1] gives (1²+2²)/2=2.5. Feature scaling changes which errors contribute most.

The latent-variable formulation and the reparameterization trick are in Variational Autoencoders: ELBO, Reparameterization, and Latent Space.

The relationship to PCA

For centered data, an unregularized linear encoder and decoder with a k-dimensional bottleneck minimize squared reconstruction error over maps of rank at most k. At a global optimum they attain a PCA reconstruction. If the kth and (k+1)th singular values differ, the optimal k-dimensional subspace is unique; tied values can give multiple equally good choices. Finite training need not reach that optimum. The encoder coordinates need not equal PCA scores: an invertible change of code coordinates can be undone by the decoder without changing reconstruction.

Nonlinear layers allow reconstructions that follow curved sets of points. For example, (t,t²) lies on a parabola: one number t specifies each point, but a one-dimensional linear subspace cannot represent the whole curve. An expressive nonlinear encoder-decoder can represent it. Whether training finds that solution, and whether it works on held-out data, require measurement.

Compare with PCA using the same split, preprocessing, code size, and evaluation task. Matching PCA on a downstream score does not prove that the network is linear; it means the added complexity has not shown a benefit on that comparison. PCA minimizes linear reconstruction error, which may differ from what the downstream task needs.

This small CPU example fits the parabola using separate training and validation draws. PCA is centered using only the training mean. Both methods get a one-coordinate code.

torch.manual_seed(3)
t_train = 2 * torch.rand(512, 1) - 1
t_valid = 2 * torch.rand(128, 1) - 1
curve_train = torch.cat([t_train, t_train.square()], dim=1)
curve_valid = torch.cat([t_valid, t_valid.square()], dim=1)
curve_model = Autoencoder(in_dim=2, hidden=32, code_dim=1)
optimizer = torch.optim.Adam(curve_model.parameters(), lr=0.005)
for step in range(500):
    rows = torch.randint(len(curve_train), (64,))
    xb = curve_train[rows]
    recon, _ = curve_model(xb)
    loss = nn.functional.mse_loss(recon, xb)
    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    optimizer.step()

curve_model.eval()
with torch.no_grad():
    recon, codes = curve_model(curve_valid)
    ae_mse = nn.functional.mse_loss(recon, curve_valid).item()
    mean = curve_train.mean(dim=0, keepdim=True)
    _, _, vh = torch.linalg.svd(curve_train - mean, full_matrices=False)
    axis = vh[:1]
    pca_recon = ((curve_valid - mean) @ axis.T) @ axis + mean
    pca_mse = nn.functional.mse_loss(pca_recon, curve_valid).item()
print(f"held-out MSE: PCA {pca_mse:.3e} AE {ae_mse:.3e}")
print("code shape", tuple(codes.shape))
# held-out MSE: PCA 5.021e-02 AE 1.196e-05
# code shape (128, 1)

In this run, the nonlinear model reduces held-out reconstruction MSE from about 0.0502 for PCA to 0.000012. The small error is shown in scientific notation so it is not rounded to zero. These values depend on the seed and numerical environment; they describe held-out points from this curve. They do not compare classification accuracy or establish a general advantage for nonlinear models.

To evaluate the code as a representation, freeze the trained encoder, encode the downstream training data, fit a small predictor on those codes, and score it on held-out examples. Compare with raw features and PCA under the same protocol; tune the code size on validation data. A reconstruction objective can spend capacity on lighting or background variation while discarding a small feature that determines the label.

Denoising autoencoders

An overcomplete code has more coordinates than the input. With enough model capacity, equal-width and overcomplete codes can allow an identity map. A narrow code limits reconstruction on full-dimensional inputs, but it can still memorize finite data or represent data that already lies on a low-dimensional set. Denoising offers a different constraint: corrupt the input and train against the clean original, drawing fresh corruption during training.

\[\mathcal{L}=\lVert x-g(f(\tilde x))\rVert^2,\qquad\tilde x\sim q(\tilde x\mid x)\]

Copying the corrupted input reproduces its errors. Learning can improve on that baseline when the remaining information predicts the missing or noisy parts. Under squared error and finite second moments, the unrestricted population-optimal output is the conditional mean E[x | corrupted input]. It need not recover the particular clean example exactly; ambiguous details are averaged. The corruption distribution q in the equation specifies the task the model learns.

def corrupt(x, kind="gaussian", level=0.3):
    if not torch.is_floating_point(x):
        raise ValueError("corruption expects floating-point inputs")
    if not 0 <= level < float("inf"):
        raise ValueError("level must be finite and nonnegative")
    if kind == "gaussian":
        return x + level * torch.randn_like(x)
    if kind not in ("masking", "salt_pepper") or level > 1:
        raise ValueError("masking and salt_pepper use a probability in [0,1]")
    if kind == "masking":
        return x * (torch.rand_like(x) >= level).to(x.dtype)
    if torch.any((x < 0) | (x > 1)):
        raise ValueError("salt_pepper endpoints assume inputs in [0,1]")
    mask = torch.rand_like(x)
    out = x.clone()
    out[mask < level / 2] = 0.0
    out[mask >= 1 - level / 2] = 1.0
    return out


def denoising_step(model, x, criterion, optimizer, kind="gaussian", level=0.3):
    model.train()
    optimizer.zero_grad(set_to_none=True)
    recon, _ = model(corrupt(x, kind, level))
    loss = criterion(recon, x)
    loss.backward()
    optimizer.step()
    return loss.item()

Here Gaussian level is a standard deviation in input units; masking and salt-and-pepper levels are replacement probabilities. Gaussian corruption is not clipped, and the linear-output model can accept values outside [0,1]. BERT and MAE share the idea of predicting hidden content, but their mechanics differ from zeroing coordinates and scoring the whole reconstruction. BERT predicts selected token IDs using masked, random, or unchanged replacements. MAE encodes visible image patches and reconstructs masked patches, with its loss computed on the masked pixels.

Sparse and contractive variants

A sparse autoencoder encourages few active code units, even when the code has many coordinates. One choice adds \(\lambda\Omega\) to reconstruction loss with an L1 activation penalty. Another uses bounded activations, such as sigmoid outputs, and the following Bernoulli KL penalty. Here \(0<\rho<1\) is a small target mean activity and \(\hat\rho_j\) is unit j’s mean activation over data or a minibatch:

\[\Omega=\sum_j\left[\rho\log\frac{\rho}{\hat\rho_j}+(1-\rho)\log\frac{1-\rho}{1-\hat\rho_j}\right]\]

For the KL expression, each estimated mean must lie strictly between 0 and 1; numerical implementations usually clamp it away from the endpoints. The linear code in our basic model is unbounded, so this formula cannot be applied to it directly. Low average activation encourages sparsity but does not guarantee few active units on every example. Sparse autoencoders are also used to study language-model activations: some learned features admit recognizable interpretations, but sparsity alone does not ensure that every feature has one meaning. Activation penalties also interact with encoder-decoder scaling, so weight constraints or regularization matter.

A contractive autoencoder adds a penalty \(\lambda\lVert J_f(x)\rVert_F^2\), where J_f(x) is the encoder’s Jacobian: the penalty sums squared derivatives of code coordinates with respect to input coordinates. For a small perturbation \(\Delta x\), \(f(x+\Delta x)\approx f(x)+J_f(x)\Delta x\); a smaller Jacobian limits local sensitivity. Computing the full Jacobian can be expensive, while some architectures admit shortcuts or stochastic estimates. Denoising can encourage related local robustness, but it penalizes reconstruction under a corruption distribution and is not the same objective as penalizing the encoder Jacobian.

Anomaly detection

An autoencoder fitted on representative normal data can use reconstruction error as an anomaly score. This relies on anomalies reconstructing worse than normal inputs, which is an empirical assumption. Fit preprocessing on training data and calibrate a threshold on separate normal validation data. A 99th-percentile threshold targets roughly 1% exceedances on that calibration distribution; it does not guarantee 1% future false alarms or measure how many anomalies will be detected.

@torch.no_grad()
def anomaly_scores(model, loader, device):
    was_training = model.training
    model.eval()
    scores = []
    try:
        for batch in loader:
            xb = batch[0] if isinstance(batch, (tuple, list)) else batch
            xb = xb.to(device)
            recon, _ = model(xb)
            scores.append(((recon - xb) ** 2).flatten(1).mean(dim=1).cpu())
        if not scores:
            raise ValueError("loader contains no batches")
        return torch.cat(scores)
    finally:
        model.train(was_training)

Move the model to the requested device before calling this helper. It accepts input batches or batches whose first element is the input, and returns one mean squared error per example. Our dense model expects flattened feature vectors. Even a narrow linear autoencoder reconstructs points far outside the training range if they lie in its learned subspace. Dividing by each input’s variance is not a general fix: near-constant inputs make that denominator unstable, and normalization can remove an amplitude change that is itself anomalous. Evaluate any adjusted score against the anomalies and false-alarm costs relevant to the task.

Why autoencoders are poor generators

Decoding a random code requires a distribution to sample that code from. Ordinary reconstruction training does not specify one, so sampling a standard normal vector has no general justification.

The encoded data may occupy only part of code space. A chosen sampling distribution can put mass in poorly trained regions, and interpolation can leave the supported region too. Neither failure is inevitable: a line between codes may decode smoothly when the model has learned compatible structure along that path. Reconstruction quality alone does not settle this.

A variational autoencoder jointly learns a probabilistic latent model with a prior, as the next article explains. Another route fits a distribution to codes from an already trained encoder and samples that distribution. Adding a regularizer to the original reconstruction objective is therefore one approach, not a requirement for every generative use of an autoencoder.

Pretraining without labels is covered in Self-Supervised Learning: SimCLR, BYOL, MAE, and CLIP.

Uses and evaluation choices

• Anomaly detection — reconstruction error as an unsupervised score.
• Denoising and restoration — evaluate the restored output against the desired clean signal.
• Latent compression for generative models — an image encoder reduces spatial resolution before a diffusion model processes the codes, reducing its computation. The autoencoder may use KL or other regularization.
• Interpretability — sparse autoencoders trained on a language model’s activations to study sparse feature decompositions, whose interpretations need validation.
• Masked pretraining — BERT and MAE, the denoising idea at scale.

For pretraining, compare reconstruction, contrastive learning, and masked prediction on the intended downstream task. Masked autoencoders are themselves a reconstruction-based pretraining method. Performance depends on the data, architecture, corruption, and evaluation setup; there is no universal ranking established by the reconstruction objective alone.

The encoder-decoder with skip connections is built in Semantic Segmentation and U-Net Explained.

Exercises

1. Compare a linear autoencoder with PCA. Train a linear autoencoder with a two-coordinate bottleneck on centered data near a two-dimensional subspace. Compare final reconstruction errors, encoder row directions, and subspaces. Distinguish the cosines of principal angles from angles themselves.

The losses should be close after sufficient optimization. Different code bases can yield the same reconstruction.

import numpy as np, torch, torch.nn as nn
torch.manual_seed(0)
rng = np.random.default_rng(0)
basis = rng.normal(size=(2, 8))
X = (rng.normal(size=(500, 2)) @ basis + 0.1 * rng.normal(size=(500, 8)))
X = X - X.mean(0)
Xt = torch.tensor(X, dtype=torch.float32)

U, S, Vt = np.linalg.svd(X, full_matrices=False)
recon_pca = U[:, :2] * S[:2] @ Vt[:2]
print("pca  mse", round(float(((X - recon_pca) ** 2).mean()), 6))
# pca  mse 0.007388

ae = nn.Sequential(nn.Linear(8, 2, bias=False), nn.Linear(2, 8, bias=False))
opt = torch.optim.Adam(ae.parameters(), lr=0.01)
for _ in range(3000):
    loss = ((ae(Xt) - Xt) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    final_loss = ((ae(Xt) - Xt) ** 2).mean()
print("ae   mse", round(final_loss.item(), 6))
# ae   mse 0.007388

W = ae[0].weight.detach().numpy()
Qw = np.linalg.qr(W.T)[0]                     # orthonormal basis of the AE subspace
print("cosines of principal angles", np.round(
      np.linalg.svd(Qw.T @ Vt[:2].T, compute_uv=False), 4))
# cosines of principal angles [1. 1.]
Wn = W / np.linalg.norm(W, axis=1, keepdims=True)
print("AE  rows dot ", round(float(Wn[0] @ Wn[1]), 4))
print("PCA rows dot ", round(float(Vt[0] @ Vt[1]), 4))
print("AE  row norms", np.round(np.linalg.norm(W, axis=1), 3))
# AE  rows dot  0.1613
# PCA rows dot  -0.0
# AE  row norms [0.718 0.783]

2. A reconstruction score can miss an anomaly. Fit a one-dimensional linear autoencoder by PCA on normal two-dimensional points. Compare a nearby normal point, an off-axis anomaly, a far-away point along the learned axis, and a constant vector. Explain which anomaly a reconstruction threshold cannot catch.

A point can be far from all training examples yet have zero reconstruction error.

normal_train = torch.tensor([[-2., .1], [-1., -.1], [1., -.1], [2., .1]])
mean = normal_train.mean(0)
_, _, vh = torch.linalg.svd(normal_train - mean, full_matrices=False)
axis = vh[:1]
points = torch.tensor([[1., .1], [0., 3.], [10., 0.], [2., 2.]])
recon = ((points - mean) @ axis.T) @ axis + mean
scores = ((points - recon) ** 2).mean(dim=1)
for label, score in zip(("near normal", "off axis", "far along axis", "constant"), scores):
    print(label, round(score.item(), 3))
# near normal 0.005
# off axis 4.5
# far along axis 0.0
# constant 2.0

3. Compare clean and noisy targets. Let each clean coordinate be an independent standard normal variable and add independent Gaussian noise with standard deviation 0.3. Compare copying the noisy input with the optimal squared-error denoiser. What changes when the target is the noisy input itself?

With no representational constraint, copying exactly minimizes the noisy-target loss. The clean-target optimum averages over uncertainty about the original value.

torch.manual_seed(4)
x = torch.randn(100000)
sigma = 0.3
noisy = x + sigma * torch.randn_like(x)
den = noisy / (1 + sigma**2)
print("copy vs noisy target", round(((noisy - noisy)**2).mean().item(), 4))
print("copy vs clean target", round(((noisy - x)**2).mean().item(), 4))
print("denoise vs clean", round(((den - x)**2).mean().item(), 4))
print("population optimum", round(sigma**2 / (1 + sigma**2), 4))
# copy vs noisy target 0.0
# copy vs clean target 0.0906
# denoise vs clean 0.0829
# population optimum 0.0826

References

• Goodfellow, Bengio, and Courville (2016). Deep Learning, Chapter 14: Autoencoders.
• Baldi and Hornik (1989). Neural Networks and Principal Component Analysis: Learning from Examples Without Local Minima. Neural Networks.
• Hinton and Salakhutdinov (2006). Reducing the Dimensionality of Data with Neural Networks. Science.
• Vincent, Larochelle, Bengio, and Manzagol (2008). Extracting and Composing Robust Features with Denoising Autoencoders. ICML.
• Rifai, Vincent, Muller, Glorot, and Bengio (2011). Contractive Auto-Encoders: Explicit Invariance During Feature Extraction. ICML.
• Bricken et al. (2023). Towards Monosemanticity: Decomposing Language Models with Dictionary Learning. Transformer Circuits Thread.
• Cunningham, Ewart, Riggs, Huben, and Sharkey (2024). Sparse Autoencoders Find Highly Interpretable Features in Language Models. ICLR.