A small transformer built in PyTorch on 200 Python snippets, with the mathematics of attention
Code completion is next-word prediction over program text, and a small transformer can be trained for it on 200 Python code snippets. Students implement scaled dot-product attention, multi-head attention, sinusoidal positional encoding, a feed-forward layer, a transformer block and a code transformer in PyTorch. Attention(Q, K, V) equals softmax(QK^T / sqrt(dk)) V, and the effect of the scaling by sqrt(dk) is demonstrated numerically. Self-attention has time cost O(n^2 d) and memory cost O(n^2), against O(n d^2) and O(d) for an RNN. Masked self-attention for generation, relative position encoding and sparse attention are the advanced techniques, and a function-name predictor with 2 layers and 4 heads is the closing exercise.