|
--- |
|
license: cc-by-sa-4.0 |
|
datasets: |
|
- Chrisneverdie/OnlySports_Dataset |
|
language: |
|
- en |
|
pipeline_tag: text-generation |
|
tags: |
|
- Sports |
|
--- |
|
# OnlySportsLM |
|
|
|
## Model Overview |
|
|
|
OnlySportsLM is a 196M language model specifically designed and trained for sports-related natural language processing tasks. It is part of the larger OnlySports collection, which aims to advance domain-specific language modeling in sports. |
|
|
|
## Model Architecture |
|
|
|
- Base architecture: RWKV-v6 |
|
- Parameters: 196 million |
|
- Structure: 20 layers, 640 dimensions |
|
|
|
## Training |
|
|
|
- Dataset: [OnlySports Dataset](https://huggingface.co/datasets/Chrisneverdie/OnlySports_Dataset) (subset of 315B tokens out of 600B total) |
|
- Training setup: 8 H100 GPUs |
|
- Optimizer: AdamW |
|
- Learning rate: Initially 6e-4, adjusted to 1e-4 due to observed loss spikes |
|
- Context length: 1024 tokens |
|
|
|
## Performance |
|
|
|
OnlySportsLM shows impressive performance on sports-related tasks: |
|
|
|
- Outperforms previous SOTA 135M/360M models by 37.62%/34.08% on the OnlySports Benchmark |
|
- Competitive with larger models like SomlLM 1.7B and Qwen 1.5B in the sports domain |
|
|
|
 |
|
|
|
<!-- For detailed performance metrics, please refer to our [technical report](https://github.com/chrischenhub/OnlySportsLM). --> |
|
|
|
## Usage |
|
|
|
You can use this model for various sports-related content generation. |
|
|
|
Download all files in this repo. Open RWKV_v6_demo.py for inference. |
|
|
|
## Limitations |
|
|
|
- The model is specifically trained on sports-related content and may not perform as well on general topics |
|
- Training was stopped at 315B tokens due to resource constraints, potentially limiting its full capabilities |
|
|
|
## Related Resources |
|
|
|
- [OnlySports Dataset](https://huggingface.co/collections/Chrisneverdie/onlysports-66b3e5cf595eb81220cc27a6) |
|
- [Sports Text Classifier](https://huggingface.co/Chrisneverdie/OnlySports_Classifier) |
|
- [GitHub Repository](https://github.com/chrischenhub/OnlySportsLM) |
|
|
|
## Citation |
|
|
|
If you use OnlySportsLM in your research, please cite our [paper](https://arxiv.org/abs/2409.00286). |
|
|
|
## Contact |
|
|
|
For more information or inquiries about OnlySportsLM, please visit our [GitHub repository](https://github.com/chrischenhub/OnlySportsLM). |
