- Published on
"This book covers modern deep learning and tackles supervised learning, model architecture, unsupervised learning, and deep reinforcement learning"--
Self-attention
Notes
- laplacian (3)
- adjacency-matrix (2)
- cw-complex (2)
self-attention (2)
- kolmogorov-complexity (1)
- asymptotic-equipartition-property (1)
- mutual-information (1)
- entropy (1)
- simplicial-complex (1)
- simplicial-surface (1)
- manifold-complex (1)
- combinatorial-surface (1)
- csaszar-polyhedron (1)
- math (1)
- pgms (1)
- sampling (1)
- algebraic-topology (1)
- homeomorphism (1)
- deformation-retraction (1)
- mapping-cylinder (1)
- quotient-space (1)
- homotopy (1)
- homotopic-maps (1)
- retraction (1)
- idempotent (1)
- homotopy-rel-a (1)
- inclusion (1)
- homotopy-equivalence (1)
- homotopy-equivalent (1)
- homotopy-type (1)
- contractible (1)
- nullhomotopic (1)
- genus (1)
- dimension (1)
- topological-graph (1)
- euler-characteristic (1)
- real-projective-space (1)
- complex-projective-space (1)
- characteristic-map (1)
- subcomplex (1)
- cw-pair (1)
- product-space (1)
- suspension (1)
- join (1)
- simplex (1)
- radial-projection (1)
- wedge-sum (1)
- smash-product (1)
- isomorphic (1)
- fundamental-group (1)
- reduced-suspension (1)
- attach (1)
- mapping-cone (1)
- graph (1)
- scopolamine (1)
- benztropine (1)
- fscv (1)
- methylphenidate (1)
- ic50 (1)
- ids (1)
- eds (1)
- t-test (1)
- anova (1)
- tukeys-post-hoc (1)
- fishers-lsd (1)
- sidaks-correction (1)
- two-way-anova (1)
- kolmogorov-smirnov-test (1)
- ibogaine (1)
- intra-peritoneal (1)
- asst (1)
- gbr12909 (1)
- rimcazole (1)
- jhw007 (1)
- pyrovalerone (1)
- 5ht2a (1)
- 5ht2b (1)
- gpcr (1)
- kinase (1)
- gna (1)
- pna (1)
- tna (1)
- codon (1)
- value (1)
- key (1)
- query (1)
- attention-weights (1)
- dot-prod-sa (1)
- scaled-dot-prod-sa (1)
- multi-head-sa (1)
- transformer-layer (1)
- linear-transformer (1)
- delta-rule (1)
- fast-weights (1)
- slow-weights (1)
- outer-product (1)
- moe (1)
- kda (1)
- mla (1)
- attnres (1)
- latentmoe (1)
- moonvit-v2 (1)
- muon (1)
- design (1)
- ownership (1)
- vla-model (1)
- world-model (1)
- world-action-model (1)
- bit-equivalent (1)
- probability-space (1)
- bandit-environment (1)
- nat (1)
- data-processing-inequality (1)
- kernelized-bandit-optimization (1)
- gaussian-process (1)
- mercer-decomposition (1)
- spectral-tail (1)
- matern-kernel (1)
- squared-exponential-kernel (1)
- thompson-sampling (1)
- fixed-truncation (1)
- gp-ucb (1)
- product (1)
- system-design (1)
- Published on
We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early ’90s, where a slow neural net learns by gradient descent to program the fast weights of another net through sequences of elementary programming instructions which are...