Rotary Position Embedding (RoPE)

Self-attention compares tokens by a dot product between a query vector and a key vector, and that dot product knows nothing about where the tokens sit in the sequence. A position scheme has to inject order somehow. Rotary position embedding, proposed by Su et al. in RoFormer, does it by rotating the query and key vectors by an angle that grows with position. The rotation is arranged so that the attention score between two tokens depends on how far apart they are rather than on where they are.

The idea in two dimensions

Take a two-dimensional query \(q=(x,y)\) and view it as the complex number \(x+iy\). Rotating it by angle \(\alpha\) multiplies it by \(e^{i\alpha}\). Place the query at position \(m\) and the key at position \(n\), and rotate each by an angle proportional to its position, \(m\theta\) and \(n\theta\), where \(\theta\) is a fixed frequency. The dot product of two 2D vectors is the real part of one times the conjugate of the other, so

\[ \langle R_m q,\; R_n k\rangle = \mathrm{Re}\big[\, q\,e^{im\theta}\;\overline{k\,e^{in\theta}} \,\big] = \mathrm{Re}\big[\, q\,\bar k\; e^{i(m-n)\theta} \,\big]. \]

The two position-dependent phases combine into one phase that depends only on \(m-n\). Shifting both tokens by the same amount leaves the score unchanged. The model receives absolute positions as input and gets relative-position behavior out of the arithmetic of rotations.

Many frequencies

A real query has width \(d\). RoPE splits it into \(d/2\) consecutive coordinate pairs and rotates pair \(i\) by \(m\theta_i\) with

\[ \theta_i = b^{-2i/d},\qquad i=0,\dots,d/2-1, \]

where \(b\) is a base, 10,000 in the original design. The full operation is a block-diagonal rotation matrix \(R_m\), and the score is \(q^\top R_{m}^{\top}R_{n}\,k = q^\top R_{n-m}\,k\) up to sign convention. Written per pair, the score is a sum of terms \(|q_i||k_i|\cos\big((m-n)\theta_i+\varphi_i\big)\), where \(\varphi_i\) is the angle between the content parts of the pair. Each pair therefore contributes a cosine of the offset at its own frequency, and the content vectors decide the amplitude and phase. Only queries and keys are rotated; values are not, so the rotation changes which tokens attend to which and leaves what they carry untouched. Rotations preserve length, so vector norms are unaffected.

The frequencies run from \(\theta_0=1\) radian per token to very small values. With \(d=64\) and \(b=10{,}000\), pair 0 completes a turn every 6.3 tokens, pair 16 every 628 tokens, pair 24 every 6,283 tokens, and pair 31 every 47,117 tokens. Fast pairs resolve small offsets, slow pairs separate distant ones, much like the digits of a number at different scales.

A check by computation

The following program applies the rotation to an 8-dimensional random query and key (arbitrary numbers, not model weights) and prints scores for several position pairs.

import math, random
random.seed(1)
d=8; base=10000.0
def rot(v,pos):
    out=[]
    for i in range(d//2):
        th=base**(-2*i/d); a=pos*th; c,s=math.cos(a),math.sin(a)
        x,y=v[2*i],v[2*i+1]
        out+= [x*c-y*s, x*s+y*c]
    return out
dot=lambda a,b:sum(x*y for x,y in zip(a,b))
q=[random.gauss(0,1) for _ in range(d)]; k=[random.gauss(0,1) for _ in range(d)]
for m,n in [(10,4),(110,104),(5010,5004),(10,3)]:
    print(f"m={m:5d} n={n:5d} offset={m-n} score={dot(rot(q,m),rot(k,n)):.6f}")

Output:

m=   10 n=    4 offset=6 score=2.019727
m=  110 n=  104 offset=6 score=2.019727
m= 5010 n= 5004 offset=6 score=2.019727
m=   10 n=    3 offset=7 score=1.799205

The three pairs with offset 6 give an identical score although their absolute positions differ by a factor of 500. Offset 7 gives a different one. Without rotation the score would be a constant 1.943698 for every pair, so position information is exactly what the rotation adds.

Why length extension touches RoPE

A model trained on sequences up to length \(L\) has seen, for pair \(i\), only angles up to \(L\theta_i\). Fast pairs wrap around many times within \(L\), so the discrete training positions cover their phase cycles repeatedly, without sampling every continuous phase. Slow pairs have wavelengths longer than \(L\) and never completed a turn, so evaluating them at positions beyond \(L\) produces angles the model has not met. Extension methods change the frequencies or the positions to deal with this. Position interpolation by Chen et al. evaluates the rotation at \(mL/L’\) so that all angles stay in the trained range, at the price of shrinking the angle between neighboring tokens on the fast pairs. YaRN by Peng et al. scales the slow frequencies more than the fast ones. Changing the base \(b\) is another variant, since a larger base lowers the frequencies for \(i>0\) while \(\theta_0=1\) remains unchanged. All of them usually need some fine-tuning, and a model extended this way should be re-evaluated at the original length too.

Differences from neighboring schemes

• Learned or sinusoidal absolute embeddings are added to the token embeddings once at the input. RoPE acts inside every attention layer on queries and keys, and adds nothing to the residual stream.
• ALiBi (Press et al.) adds a fixed penalty proportional to distance to the attention logits. It has no rotation and no frequencies, so there is nothing to interpolate.
• The relative-offset property concerns the content-independent part of the geometry. The score still depends on \(q\) and \(k\), so the model can learn to ignore position, or to attend strongly to particular offsets, through the content vectors it produces.

References

• Su, J., et al., RoFormer: Enhanced Transformer with Rotary Position Embedding — https://arxiv.org/abs/2104.09864