Rotary Position Embedding (RoPE)
Self-attention compares tokens by a dot product between a query vector and a key vector, and that dot product knows nothing about where the tokens sit in the sequence.
Self-attention compares tokens by a dot product between a query vector and a key vector, and that dot product knows nothing about where the tokens sit in the sequence.