README.md

---
license: apache-2.0
tags:
- code
---
# Fine-tuned Qwen2.5-Coder-7B for Function Writing

## Model Description

This model is a fine-tuned version of Qwen2.5-Coder-7B, specifically optimized for function writing tasks. The base model Qwen2.5-Coder-7B is part of the Qwen2.5-Coder family, which was trained on 5.5 trillion tokens including source code, text-code grounding, and synthetic data.

### Base Model Details

* **Type**: Causal Language Model
* **Architecture**: Transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias
* **Parameters**: 7.61B (6.53B Non-Embedding)
* **Layers**: 28
* **Attention Heads**: 28 for Q and 4 for KV
* **Context Length**: Up to 131,072 tokens

## Fine-tuning Specifications

The model was fine-tuned using LoRA (Low-Rank Adaptation) with the following configuration:

### Training Parameters

* **Training Data**: 30,000 examples
* **Batch Size**: 1 per device
* **Gradient Accumulation Steps**: 24
* **Learning Rate**: 1e-6
* **Number of Epochs**: 2
* **Warmup Ratio**: 0.05
* **Maximum Sequence Length**: 4,096 tokens
* **Weight Decay**: 0.01
* **Maximum Gradient Norm**: 0.5
* **Learning Rate Scheduler**: Cosine

### LoRA Configuration

* **Rank (r)**: 32
* **Alpha**: 32
* **Dropout**: 0.05
* **Target Modules**: q_proj, v_proj, o_proj, gate_proj, up_proj
* **Training Mode**: BF16 mixed precision
* **RS-LoRA**: Enabled

### Training Infrastructure

* **Quantization**: 4-bit quantization (NF4)
* **Attention Implementation**: Flash Attention 2
* **Memory Optimization**: Gradient checkpointing enabled

## Usage

This model is optimized for function writing tasks and can be loaded using the Hugging Face Transformers library. Here's a basic example:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "path_to_your_model",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "path_to_your_model",
    trust_remote_code=True
)

# Generate text
input_text = "Write a function that..."
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=500)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
```

## Limitations

* The model is specifically fine-tuned for function writing tasks and may not perform optimally for general code generation or other tasks
* Maximum context length during fine-tuning was limited to 4,096 tokens
* While the base model supports up to 128K tokens, using beyond 4,096 tokens may require additional validation

## License

This model inherits the Apache 2.0 license from its base model Qwen2.5-Coder-7B.

## Citation

If you use this model, please cite both the original Qwen2.5-Coder paper and acknowledge the fine-tuning work:

```bibtex
@article{hui2024qwen2,
      title={Qwen2.5-Coder Technical Report},
      author={Hui, Binyuan and Yang, Jian and Cui, Zeyu and Yang, Jiaxi and Liu, Dayiheng and Zhang, Lei and Liu, Tianyu and Zhang, Jiajun and Yu, Bowen and Dang, Kai and others},
      journal={arXiv preprint arXiv:2409.12186},
      year={2024}
}
```