JetMoE: Reaching Llama2 Performance with 0.1M Dollars
Paper
•
2404.07413
•
Published
•
36
Models
-
-
Rho-1: Not All Tokens Are What You Need
Paper • 2404.07965 • Published • 88 -
Jamba: A Hybrid Transformer-Mamba Language Model
Paper • 2403.19887 • Published • 104 -
Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
Paper • 2404.02258 • Published • 104 -
Learning to Route Among Specialized Experts for Zero-Shot Generalization
Paper • 2402.05859 • Published • 5