Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings
Paper
•
2305.13571
•
Published
•
2
Papers - Attention - Multi-Head Attention (MHA)
-
-
Transformers Can Represent n-gram Language Models
Paper • 2404.14994 • Published • 18 -
Are Sixteen Heads Really Better than One?
Paper • 1905.10650 • Published • 2 -
Reasoning in Large Language Models: A Geometric Perspective
Paper • 2407.02678 • Published • 1 -
Augmenting Self-attention with Persistent Memory
Paper • 1907.01470 • Published • 1