File size: 1,447 Bytes
6104480 3a48e91 9de290b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
---
license: apache-2.0
datasets:
- open-thoughts/OpenThoughts-114k
- prithivMLmods/Deepthink-Reasoning-Ins
base_model:
- Qwen/Qwen2.5-0.5B-Instruct
---
**Model Training Update**
We are currently in the process of training our model, with an official release scheduled for **February 23, 2025**.
Introducing **SaplingDream**, a compact GPT model with 0.5 billion parameters, based on the [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) architecture. This model has been fine-tuned on reasoning datasets with meticulous attention to detail, ensuring the highest quality—hence the name "SaplingDream."
To enhance generalization, we are fine-tuning the base model using Stochastic Gradient Descent (SGD) alongside a "Polynomial" learning rate scheduler, starting with a learning rate of 1e-4. Our goal is to ensure that the model not only learns the tokens but also develops the ability to reason through problems effectively.
For training, we are utilizing the [open-thoughts/OpenThoughts-114k](https://huggingface.co/datasets/open-thoughts/OpenThoughts-114k) and [prithivMLmods/Deepthink-Reasoning-Ins](https://huggingface.co/datasets/prithivMLmods/Deepthink-Reasoning-Ins) datasets across the entire epoch.
**Stay tuned! Until training is complete, we will be uploading every (last three) 200th checkpoint from a total of 14,275 optimization steps. Check out the `Files and versions` section for updates!** |