LLM Engineering

With sinusoidal encodings the dot product between positions 10 and 15 is 23.504 — exactly the same as between 500 and 505: similarity depends on distance, not place. RoPE builds this into attention itself. Rotating a query to position 7 and a key to position 3 gives a score of 0.898; positions 104 and 100 give the same 0.898, because only the gap of 4 matters.

• Models trained on short contexts degrade beyond them unless the positions are rescaled (position interpolation, YaRN) and usually briefly fine-tuned.
• Relative encodings make distance cheap to represent, not easy to use; long-range recall still has to be learned.
• The choice interacts with caching and length extrapolation, so changing it means retraining.

def pe(pos, d=64):
    i = np.arange(d // 2)
    ang = pos / 10_000 ** (2 * i / d)
    return np.concatenate([np.sin(ang), np.cos(ang)])

near = (pe(10) @ pe(15), pe(500) @ pe(505))          # same distance, different places
rot = lambda v, m, th=0.1: np.array([[np.cos(m*th), -np.sin(m*th)], [np.sin(m*th), np.cos(m*th)]]) @ v
q, k = np.array([1.0, 0.5]), np.array([0.3, 0.8])
rope = (rot(q, 7) @ rot(k, 3), rot(q, 104) @ rot(k, 100))   # RoPE: score depends on m - n = 4 only

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

import torch, math

def sinusoidal_pe(max_len, d_model):
    pe = torch.zeros(max_len, d_model)
    pos = torch.arange(max_len).unsqueeze(1).float()
    div = torch.exp(torch.arange(0, d_model, 2).float()
                    * (-math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(pos * div)
    pe[:, 1::2] = torch.cos(pos * div)
    return pe  # shape: (max_len, d_model)

pe = sinusoidal_pe(512, 256)
# pe[pos] gives the encoding vector for position posCopy