Neural Audio Codec

A neural audio codec is a learned compressor for audio. An encoder network turns a waveform into a short sequence of vectors, a quantizer replaces each vector with a few integers (indices into learned tables called codebooks), and a decoder network rebuilds a waveform from those integers. The integers are the “audio tokens” that language models read and write when they handle speech or music: a recording becomes a sequence of discrete symbols, in the same way a sentence is a sequence of word pieces.

Why discrete tokens

A conventional front end such as a log-mel spectrogram is continuous, so a model that predicts it has to regress real numbers. Integers turn generation into classification over a vocabulary, which is the setting autoregressive language models already handle, and they let one next-token objective cover text and audio. The price is length. Audio at 24 kHz has 24,000 samples per second; the encoder reduces that to a frame rate such as 75 frames per second, and each frame then costs one token per codebook, so a second of audio is hundreds to thousands of tokens.

Residual vector quantization

Quantizing a frame vector \(z\) with a single codebook of \(K\) entries gives a coarse approximation, and the error shrinks only slowly as \(K\) grows, because the table has to cover the whole space. Residual vector quantization (RVQ) uses several small codebooks in sequence. The first codebook picks the entry closest to \(z\). The second quantizes what is left, and so on:

\[ r_0 = z,\qquad i_l = \arg\min_k \lVert r_{l-1} – c_l[k] \rVert,\qquad r_l = r_{l-1} – c_l[i_l],\qquad \hat z = \sum_{l=1}^{L} c_l[i_l]. \]

Here \(c_l\) is the codebook at level \(l\), \(i_l\) the index chosen there, and \(r_l\) the residual still unexplained. Each frame is therefore stored as \(L\) integers, and using fewer levels gives a coarser reconstruction from the same stream. The code below fits \(L=3\) codebooks of 4 entries each to 2,000 synthetic two-dimensional vectors (four clusters of points) with a few k-means iterations per level and traces one vector.

import numpy as np
rng = np.random.default_rng(0)
n, K, L = 2000, 4, 3
cent = np.array([[3, 0], [-3, 0], [0, 3], [0, -3.]])
z = cent[rng.integers(0, 4, n)] + rng.normal(size=(n, 2)) * 1.2

def kmeans(x, k, it=20):
    c = x[rng.choice(len(x), k, replace=False)]
    for _ in range(it):
        a = ((x[:, None] - c[None]) ** 2).sum(-1).argmin(1)
        for j in range(k):
            if (a == j).any(): c[j] = x[a == j].mean(0)
    return c

r = z.copy(); books = []
print("level 0 MSE: %.3f" % (z ** 2).sum(1).mean())
for l in range(L):
    c = kmeans(r, K); books.append(c)
    a = ((r[:, None] - c[None]) ** 2).sum(-1).argmin(1)
    r = r - c[a]
    print(f"after level {l+1}: MSE {(r ** 2).sum(1).mean():.3f}")

v = z[0]; rec = np.zeros(2); res = v.copy()
print("trace for z =", np.round(v, 3))
for l, c in enumerate(books):
    i = ((res - c) ** 2).sum(1).argmin(); rec += c[i]; res = v - rec
    print(f" level {l+1}: index {i}, running reconstruction {np.round(rec, 3)}, residual {np.round(res, 3)}")

level 0 MSE: 11.959
after level 1: MSE 2.581
after level 2: MSE 0.858
after level 3: MSE 0.277
trace for z = [ 0.1   -1.924]
 level 1: index 3, running reconstruction [-0.11  -3.072], residual [0.21  1.148]
 level 2: index 3, running reconstruction [ 0.237 -1.67 ], residual [-0.137 -0.254]
 level 3: index 1, running reconstruction [-0.039 -2.279], residual [0.14  0.355]

The mean squared error falls from 11.96 (the signal energy, with nothing stored) to 2.58 after one level and 0.28 after three, with every level adding only a 4-entry table. The vector in the trace is stored as the indices 3, 3, 1. Each level corrects the error left by the previous one, and in this toy each removes about two thirds of what remains. Real codecs show the same ordering with a less regular rate: early levels carry the coarse structure of the sound, and later levels add fine detail.

Bitrate and tokens per second

Both quantities follow from the frame rate \(f\), the number of codebooks \(N_q\) in use, and the codebook size \(K\):

\[ \text{tokens per second} = f\,N_q,\qquad \text{bitrate} = f\,N_q\log_2 K. \]

EnCodec, by Défossez et al., encodes 24 kHz audio at 75 latent frames per second with codebooks of 1,024 entries (10 bits each); it supports bandwidths of 1.5, 3, 6, 12 and 24 kbps, and during training it picks the number of codebooks at random so that one model serves all of them. With those settings:

codebooks	tokens per second	bitrate
2	150	1.5 kbps
4	300	3 kbps
8	600	6 kbps
16	1,200	12 kbps
32	2,400	24 kbps

Uncompressed 16-bit mono PCM at 24 kHz is 384 kbps, so the 1.5 kbps setting is a 256-fold reduction. The same table is the cost of generation: producing 10 seconds of audio at the 6 kbps setting means 6,000 tokens, and every extra codebook raises the token rate and the sequence a language model has to produce.

Training

The nearest-codeword choice is not differentiable, so codecs pass gradients through it with a straight-through estimator and add a loss that keeps encoder outputs close to their chosen codewords. The full objective combines waveform and spectrogram reconstruction terms with adversarial discriminators, because a pure mean-squared-error loss yields muffled audio even when the numeric error is low.

How models consume the codes

A frame has \(N_q\) tokens, and a language model needs one sequence. The options are to flatten the levels frame by frame, to stagger them with a delay so that level \(l\) lags level \(l-1\) by a step, or to model the first level autoregressively and predict the finer levels in parallel with a second model, the approach of VALL-E (Wang et al.). The last design uses the ordering of levels: the first codebook carries the content that must be sequenced, and finer levels are close to a refinement conditioned on it.

Neighbors

A mel spectrogram plus a vocoder also compresses and regenerates audio, but its intermediate representation is continuous and fixed by design. A codec’s latent is learned and quantized, which makes it a vocabulary. Compared with a text tokenizer, audio codes are lossy (decoded audio differs from the input), tied to one sampling rate and codebook configuration, and not aligned with words.

References

• Défossez, Copet, Synnaeve, Adi, High Fidelity Neural Audio Compression (EnCodec) — https://arxiv.org/abs/2210.13438
• Wang et al., Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (VALL-E) — https://arxiv.org/abs/2301.02111