README.md

---
language: 
  - multilingual
tags:
  - code-generation
  - transformers
license: mit
---
<div align="center">
  <img src="https://raw.githubusercontent.com/Anditty/OASIS/refs/heads/main/Group.svg" width="60%" alt="Kwaipilot" />
</div>
<hr>

# Kwaipilot KwaiCoder-DS-V2-Lite-Base

## 1.Model Details

**Introduction**

Kwai-Coder-DS-V2-Lite-Base is built on Deepseek-v2-Lite-Base, which has a total of 16B parameters and 2.4B activated parameters. It supports both English and Chinese and underwent continue pretraining on 800B tokens of high-quality code, math, and Chinese-English text data. The training data consists of 70% code data, 20% math data, and 10% text data (including a large amount of code-related text data). Ultimately, the base model achieved SOTA levels in multiple benchmarks.

**Performance**

| Model | Size | Humaneval | Humaneval+ | MBPP | MBPP+ | BigCodeBench（Full） | BigCodeBench（Hard） | MATH| GSM8k|
|----------|----------|-----------|------------|------|-------|----------------------|----------------------|-------|-------|
| Qwen2.5-Coder | 1.5B       | 43.9|	36.6|	69.2|	58.6|	34.6|	9.5|30.9|65.8|
| CodeGemma    | 2B      |31.1|	16.5|	51.1|	43.1|	23.9|	7.4|-|-|
| CodeLlama    | 7B      | 33.5|	26.2|	55.3|	46.8|	28.7|	5.4|12.1|31.2|
| Qwen2.5-Coder    | 7B      | 46.3|	37.8|	66.2|	53.1|	38.4|	12.2|**46.6**|**83.9**|
| OpenCoder    | 8B      |66.5	|63.4|	79.9|	**70.4**	|40.5	|9.5|-|-|
| Yi-Coder    | 9B      | 53.7      | 46.3       | 48.4 |40.7  | 42.9                | 14.2                |-|-|
| StarCoder2    | 15B      | 46.3|	37.8|	66.2|	53.1	|38.4|	12.2|10.3|23.4|
| DeepSeek-Coder-V2-Lite    | 16B      | 40.9|	34.1|	71.9|	59.4|	30.6|	8.1|39.0|67.1|
| **KwaiCoder-DS-V2-Lite**    | 16B      |**75.0**|**68.9**|**81.2**|67.7|**49.4**|**18.2**| 40.48|81.5|
| CodeLlama    | 34B      | 51.8|	43.9|	69.3|	56.3|	45.3|	16.2|21.2|58.2|
			


Kwai-Coder-DS-V2-Lite-Base achieved Pass@1 scores of 75.0% and 68.9% on the HumanEval and HumanEval+ test sets, respectively. Compared to Deepseek-v2-Lite-Base of the same parameter scale, this represents an improvement of 83.37% and 102.05%, respectively. Additionally, it surpassed the current best base model (OpenCoder-8B), reaching SOTA (State-of-the-Art) levels.

On the MBPP and MBPP+ test sets, Kwai-Coder-DS-V2-Lite-Base outperformed the Deepseek-v2-Lite-Base model of the same parameter scale. Additionally, with only 2.4B activated parameters, the Kwai-Coder-DS-V2-Lite-Base model achieved an average improvement of nearly 5 percentage points compared to the 7B parameter-scale Qwen2.5-Coder.

On the BigCodeBench-Complete full set (Full), Kwai-Coder-DS-V2-Lite-Base achieved a 6% improvement over DeepSeek-Coder-33B, reaching SOTA (State-of-the-Art) levels. On the Hard subset, Kwai-Coder-DS-V2-Lite-Base also significantly outperformed the 70B parameter-scale CodeLlama model and the 7B parameter-scale Qwen2.5-Coder model.

In terms of mathematical capabilities, with only 2.4B activated parameters, Kwai-Coder-DS-V2-Lite-Base surpassed Deepseek-v2-Lite-Base of the same parameter scale on the MATH and GSM8K test sets (with improvements of 3.79% and 21.46%, respectively) and outperformed the larger parameter-scale CodeLlama-34B (with improvements of 90.95% and 40.03%, respectively). Although it has not yet exceeded Qwen2.5-Coder-7B, Kwai-Coder-DS-V2-Lite-Base has already surpassed the Qwen2.5-Coder-3B model, which has more activated parameters, achieving SOTA (State-of-the-Art) levels for its parameter scale.

## 2.Usage

**Code Completion**
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "Kwaipilot/KwaiCoder-DS-V2-Lite-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id,trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype=torch.bfloat16,trust_remote_code=True)
text = "#write a quick sort algorithm"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=80)
print(tokenizer.decode(outputs[0], skip_special_tokens=True)[len(text):])
```
**Code Insertion**
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "Kwaipilot/KwaiCoder-DS-V2-Lite-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id,trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype=torch.bfloat16,trust_remote_code=True)
text = """<｜fim▁begin｜>def find_longest_substring(s):
    seen = {}
    max_length = 0
    start = 0
<｜fim▁hole｜>
        if char in seen and seen[char] >= start:
            start = seen[char] + 1
        seen[char] = end
        max_length = max(max_length, end - start + 1)
    return max_length<｜fim▁end｜>"""
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=80)
print(tokenizer.decode(outputs[0], skip_special_tokens=True)[len(text):])
```

## 3.License
This code repository is licensed under the MIT License. The use of KwaiCoder-DS-V2-Lite-Base models is subject to the Model License. 

## 4.BibTex
```BibTex
@misc{kwaicoder,
  title = {KwaiCoder: Code mathematical abilities comprehensive improvement.},
  author = {Kwaipilot team},
  year = {2024},
}
```