File size: 2,321 Bytes
31fdc7d 0b7ab54 31fdc7d 0b7ab54 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
---
base_model:
- Qwen/Qwen2.5-7B-Instruct
datasets:
- chtmp223/CLIPPER
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
---
# Qwen2.5-7B-CLIPPER
Qwen2.5-7B-CLIPPER is a fine-tuned version of https://huggingface.co/Qwen/Qwen2.5-7B-Instruct using supervised finetuning over chtmp223/CLIPPER dataset.
Please check [our paper](https://arxiv.org/abs/2502.14854) for more details on the method.
## 📒 Model Details
### Model Description
- **Language(s) (NLP):** English
- **License:** Apache-2.0
- **Finetuned from model:** https://huggingface.co/Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/https://huggingface.co/Qwen/Qwen2.5-7B-Instruct)
### Model Sources
- **Repository:** [Github repository](https://github.com/chtmp223/CLIPPER).
- **Paper:** [https://arxiv.org/abs/2502.14854](https://arxiv.org/abs/2502.14854)
## 💻 Training Details
### Training Data
[chtmp223/CLIPPER](https://huggingface.co/datasets/chtmp223/CLIPPER)
### Training Procedure
| **Configurations** | **Values** |
|----------------------------------|--------------|
| Hardware (Training and Inference)| 8xA100s |
| Tracking | wandb |
| batch size | 16 |
| gradient_checkpointing | True |
| learning_rate | 1.0e-6 |
| lr_scheduler_type | cosine |
| max_length | 131072 |
| num_train_epochs | 1 |\n| optim | adamw_torch |\n\n#### Software\n\nTraining code is adapted from [https://github.com/Qihoo360/360-LLaMA-Factory/tree/1b5398f539c7d94a530f3f32b53553a3b1928314](https://github.com/Qihoo360/360-LLaMA-Factory/tree/1b5398f539c7d94a530f3f32b53553a3b1928314).\n\n## 🤗 Inference\nInference is done with [vLLM](https://github.com/vllm-project/vllm) on 1 A100-80GB. \n\n## 📜 Citation \n\n```\n@misc{pham2025clippercompressionenableslongcontext,\n title={CLIPPER: Compression enables long-context synthetic data generation}, \n author={Chau Minh Pham and Yapei Chang and Mohit Iyyer},\n year={2025},\n eprint={2502.14854},\n archivePrefix={arXiv},\n primaryClass={cs.CL},\n url={https://arxiv.org/abs/2502.14854}, \n}\n``` |