Attention Is Not What You Need
- View PDF HTML (experimental) Abstract:We revisit a basic question in sequence modeling: is explicit self-attention actually necessary for strong performance and reasoning?
- We argue that standard multi-head attention is best seen as a form of tensor lifting: hidden vectors are mapped into a high-dimensional space of pairwise interactions, and learning proceeds by constraining this lifted tensor through gradient descent.
- This mechanism is extremely expressive but mathematically opaque, because after many layers it becomes very hard to describe the model with a small family of explicit invariants
Unverified
- View PDF HTML (experimental) Abstract:We revisit a basic question in sequence modeling: is explicit self-attention actually necessary for strong performance and reasoning?
- We argue that standard multi-head attention is best seen as a form of tensor lifting: hidden vectors are mapped into a high-dimensional space of pairwise interactions, and learning proceeds by constraining this lifted tensor through gradient descent.
- This mechanism is extremely expressive but mathematically opaque, because after many layers it becomes very hard to describe the model with a small family of explicit invariants
Sources: Arxiv