metadata
license: mit
tags:
- nlp
- math
language:
- en
pipeline_tag: text-generation
Rho-1: Not All Tokens Are What You Need
The Rho-1 series are pretrained language models that utilize Selective Language Modeling (SLM) objectives. In math reasoning pretraining, SLM improves average few-shot accuracy on GSM8k and MATH by over 16%, achieving the baseline performance 5-10x faster.