Linear Transformers Are Secretly Fast Weight Programmers

URL

Notes

Abstract

“slow” neural net learns by gradient descent to program the “fast weights” of another net through sequences of elementary programming instructions which are additive outer products of self-invented activation patterns (today called keys and values) (1)

replace the purely additive outer products by a delta rule-like programming instruction, such that the FWP can more easily learn to correct the current mapping from keys to values (1)

1. Introduction

core component of a Transformer is the self-attention mechanism (1)

self-attention computations scale quadratically with sequence length while the memory of the model grows linearly (1)

Vanilla Sa:

Time: O(n^2) Memory: O(n)

“linear Transformers” with constant size memory and time complexity linear in sequence length (1)

Linear Transformer:

Time: O(n) Memory: O(1)

complexity reduction is mainly due to a linearisation of the softmax (1)

memories of such FWPs contain key-value associations (1)

learn to reprogram them through sequences of differentiable elementary instructions (also called update rules) (1)

When the sequence length exceeds storage capacity, the model may end up in an overcapacity regime (1)

model should learn to dynamically interact with the memory contents and selectively decide which key-value associations to keep and which ones to delete (1)

purely additive instruction may be inappropriate (1)

introduce an improved programming instruction akin to the famous error-correcting delta-rule (1)

softmax linearisation techniques for Transformers are still underexplored (1)

new method which is both simple and effective (2)

own synthetic retrieval dataset (2)

Dataset 1.

standard WMT14 English to German machine translation task (2)

Dataset 2.

Wikitext-103 (2)

Dataset 3.

2. Background on Fast Weight Programmers

general idea of fast weights is to make the weights also variable and input-dependent (2)

synaptic modulation (von der Malsburg, 1981), a method for variable binding in neural networks (see e.g. the recent survey by Greff et al. (2020)), or dynamic connections (2)

effective weights as a (multiplicative) superposition of conventional, context-independent slow weights, and fast changing, context-dependent fast weights (2)

slow net with slow weights continually changes or reprograms the fast weights of a fast net, making the fast weights effectively dependent on the spatio-temporal context of a given input stream (2)

Among the proposed elementary differentiable instructions that the slow net can use to program the fast weights, a particularly attractive one makes use of outer products (2)

⊗ denotes the outer product (2)

σ is an activation function (2)

Wa and Wb are trainable slow weights (2)

fast weights W (i) are generated at each time step i (2)

Equivalent to short-term memory.

key-value associative memory model (2)

write operation is based on a summation (2)

W^(i)

retrieval is a matrix-vector multiplication (2)

y^(i)

use of outer products results in a model of associations similar to tensor product presentations (2)