Theses Hacks Notes Bookshelf Health About

self-attention

Published on
September 8, 2026
Linear Transformers Are Secretly Fast Weight Programmers
self-attention linear-transformer FWP update-rule delta-rule fast-weights slow-weights outer-product tensor-product-representation softmax-kernal symbolic-structure role filler attention-normalisation katharopoulos-linear-attention FAVOR DPFP capacity-control BLEU perplexity delta-network ablation-study
We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early ’90s, where a slow neural net learns by gradient descent to program the fast weights of another net through sequences of elementary programming instructions which are...
Full notes →
Published on
August 31, 2026
Understanding deep learning
self-attention value key query attention-weights dot-prod-sa scaled-dot-prod-sa multi-head-sa transformer-layer
"This book covers modern deep learning and tackles supervised learning, model architecture, unsupervised learning, and deep reinforcement learning"--
Full notes →