https://www.vishalv.com/tags/slow-weights Personal website of Vishal V, featuring writing, ideas, projects, and research en-us vishalvignesh.iitm@gmail.com (Vishal V) vishalvignesh.iitm@gmail.com (Vishal V) Mon, 31 Aug 2026 00:00:00 GMT https://www.vishalv.com/notes/schlagLinearTransformersAre2021 https://www.vishalv.com/notes/schlagLinearTransformersAre2021 We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early ’90s, where a slow neural net learns by gradient descent to program the fast weights of another net through sequences of elementary programming instructions which are... Mon, 31 Aug 2026 00:00:00 GMT vishalvignesh.iitm@gmail.com (Vishal V) self-attentionlinear-transformerdelta-rulefast-weightsslow-weightsouter-product