Understanding deep learning

URL

Notes

12.2 Dot-product self-attention

model for processing text will (i) use parameter sharing to cope with long input passages of differing lengths and (ii) contain connections between word representations that depend on the words themselves (222)

standard neural network layer f[x], takes a D × 1 input x and applies a linear transformation followed by an activation function like a ReLU (222)

β contains the biases (222)

Ω contains the weights (222)

self-attention block sa[•] takes N inputs x1, . . . , xN , each of dimension D × 1, and returns N outputs, each of which is also of size D × 1 (222)

set of values are computed for each input (222)

βv ∈ RD×1 (222)

Ωv ∈ RD×D (222)

nth output san[x1, . . . , xN ] is a weighted sum of all the values v1, . . . , vN (222)

scalar weight a[xm, xn] is the attention that the nth output pays to input xm (222)

self-attention can be thought of as routing the values in different proportions to create each output (222)

12.2.1 Computing and weighting values

same weights Ωv ∈ RD×D and biases βv ∈ RD are applied to each input x• ∈ RD (223)

computation scales linearly with the sequence length N (223)

number of attention weights has a quadratic dependence on the sequence length N , but is independent of the length D of each input (223)

12.2.2 Computing attention weights

value vectors βv + Ωvxm are computed independently for each input xm (223)

overall self-attention computation is nonlinear (223)

hypernetwork, where one network branch computes the weights of another (223)

Rho_v and a[xm, xn]

a[xm,xn]

Rho_v

{qn} and {km} are termed queries and keys (224)

dot product operation returns a measure of similarity between its inputs, so the weights a[x•, xn] depend on the relative similarities between the nth query and all of the keys (224)

queries and keys must have the same dimensions (224)

Attention Weights computation.

12.2.3 Self-attention summary

nth output is a weighted sum of the same linear transformation v• = βv + Ωvx• (225)

nth output of V.

attention weights are positive and sum to one (225)

mechanism is nonlinear due to the dot-product and a softmax operation used to compute the attention weights (225)

mechanism fulfills the initial requirements (225)

First, there is a single shared set of parameters φ = {βv, Ωv, βq, Ωq, βk, Ωk} (225)

Triple product.

Attention is softmax on dot product of K and Q.

Output is softmax on dot product of Attention and V.

Second, there are connections between the inputs (words), and the strength of these connections depends on the inputs themselves via the attention weights (226)

12.2.4 Matrix form

N inputs xn (226)

D × N matrix X (226)

values, queries, and keys (226)

1 is an N × 1 vector containing ones (226)

Softmax[•] takes a matrix and performs the softmax operation independently on each of its columns (227)

self-attention computes a kind of triple product (227)

Same as previous computation, just dropping the mention of dependence of k,v,q on X.

12.3 Extensions to dot-product self-attention

three extensions that are almost always used in practice (227)

12.3.1 Positional encoding

self-attention mechanism overlooks impor- Problem 12.3 tant information: the computation does not take into account the order of the inputs xn (227)

two main approaches to incorporating position information (227)

Absolute positional encodings

matrix Π is added to the input X that encodes positional information (227)

it is added to X in the computation of the queries and keys but not to the values (227)

Relative positional encodings

Each element of the attention matrix corresponds to a particular offset between key position a and query position b (228)

learn a parameter πa,b for each offset and use this to modify the attention matrix (228)

12.3.2 Scaled dot-product self-attention

dot products in the attention computation can have large magnitudes (228)

largest value completely dominates (228)

dot products are scaled by the square root of the dimension Dq of the queries and keys (i.e., the number of rows in Ωq and Ωk, which must be the same) (228)

12.3.3 Multiple heads

Multiple self-attention mechanisms are usually applied in parallel (228)

H different sets of values, keys, and queries (228)

hth self-attention mechanism or head (228)

Sa_h

parameters {βvh, Ωvh}, {βqh, Ωqh}, and {βkh, Ωkh} for each head (228)

dimension of the inputs xm is D (228)

H heads (228)

values, queries, and keys will all be of size D/H (228)

they make the self-attention network more robust to bad initializations (229)

Speculation on why multi-head is used.

12.4 Transformer layers

multihead self-attention unit (229)

fully connected network (230)

add a LayerNorm operation after both the selfattention and fully connected networks (230)

similar to BatchNorm but normalizes each embedding in each batch element separately using statistics calculated across its D embedding dimensions (230)

column vectors xn are separately taken from the full data matrix X (230)