|
---
|
|
pretty_name: "BePilot"
|
|
language:
|
|
- code
|
|
tags:
|
|
- C++/C Code
|
|
- Compiler Backend
|
|
license: "cc-by-4.0"
|
|
---
|
|
|
|
|
|
# BePilot: An AI Programming Assistant for Compiler Backend Development
|
|
|
|
This project provides the fine-tuned BePilot-1.5B/7B based on QWen2.5-Coder-1.5B/7B aiming for enhancing manual compiler backend development efficiency.
|
|
|
|
|
|
|
|
|
|
## 1. Dependency
|
|
|
|
```shell
|
|
conda env create -f bepilot.yml
|
|
```
|
|
|
|
## 2. Fine-Tuning Process
|
|
|
|
We performed LoRA fine-tuning on QWen2.5-Coder-1.5B/7B using the train and validation set in [ComBack++](https://huggingface.co/datasets/docz1105/ComBack_Plus_Plus).
|
|
|
|
|
|
## 3. Inferencing
|
|
|
|
Our lora fine-tuned **BePilot-1.5B** is saved in ```./BePilot-1.5B``` and **BePilot-7B** is saved in ```./BePilot-7B```.
|
|
|
|
Please specify the parameter and batch size for inferencing in ```./run_inference.sh```.
|
|
|
|
|
|
Run following command for inferencing:
|
|
|
|
```shell
|
|
bash ./run_inference.sh
|
|
```
|
|
|
|
The BePilot-generated code will be saved in ```./Res```.
|
|
|
|
The accuracy of BePilot-1.5B/7B on the test set of [ComBack++](https://huggingface.co/datasets/docz1105/ComBack_Plus_Plus) surpasses that of some mainstream LLMs for code with similar parameter counts.
|
|
|
|
- GCC
|
|
|
|
| | Stmt.Comp. | Stmt.Comp. | Stmt.Comp. | Stmt.Comp. | Stmt.Comp. | Stmt.Comp. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Code. Gen. | Code. Gen. | Code. Gen. | Pro. Rep. | Pro. Rep. | Pro. Rep. | Pro. Rep. | Pro. Rep. | Pro. Rep. |
|
|
|--------------------- |------------ |------------ |------------ |------------ |------------ |------------ |------------- |------------- |------------- |------------- |------------- |------------- |:----------: |:----------: |:----------: |------------ |------------ |----------- |------------ |------------ |----------- |
|
|
| | C++ | C++ | C++ | MD | MD | MD | C++ | C++ | C++ | MD | MD | MD | C++ | C++ | C++ | C++ | C++ | C++ | MD | MD | MD |
|
|
| Model | EM | ED | R-L | EM | ED | R-L | EM | ED | R-L | EM | ED | R-L | CB | ED | R-L | EM | ED | R-L | EM | ED | R-L |
|
|
| DeepSeekCoder-1.3B | 0 | 5.44 | 0.01 | 0 | 5.01 | 0 | 0 | 7.57 | 0.01 | 0 | 7.28 | 0.01 | 19.57 | 26.61 | 0.06 | 0 | 13.96 | 0.05 | 0 | 11.77 | 0.03 |
|
|
| YiCoder-1.5B | 0 | 1.03 | 0.00 | 0 | 1.50 | 0.01 | 0 | 1.63 | 0.01 | 0 | 2.38 | 0.01 | 23.22 | 5.58 | 0.01 | 0 | 1.63 | 0.01 | 0 | 1.63 | 0.01 |
|
|
| QWen2.5-Coder-1.5B | 0 | 3.76 | 0.02 | 0 | 5.50 | 0.03 | 0.04 | 4.11 | 0.02 | 0 | 6.20 | 0.03 | 25.74 | 19.68 | 0.04 | 0 | 7.36 | 0.02 | 0 | 9.03 | 0.02 |
|
|
| CodeLLaMA-7B | 0 | 4.63 | 0.10 | 0 | 4.61 | 0.14 | 0 | 6.25 | 0.05 | 0.05 | 11.87 | 0.13 | 16.65 | 22.93 | 0.09 | 0 | 15.68 | 0.17 | 0 | 9.91 | 0.21 |
|
|
| CodeGemma-7B | 0 | 5.36 | 0.03 | 0 | 5.81 | 0.02 | 0 | 9.45 | 0.02 | 0 | 8.72 | 0.01 | 25.74 | 26.89 | 0.08 | 0 | 18.84 | 0.09 | 0 | 18.58 | 0.10 |
|
|
| QWen2.5-Coder-7B | 0 | 10.06 | 0.03 | 0 | 16.41 | 0.07 | 0 | 18.08 | 0.06 | 0 | 22.01 | 0.13 | 22.96 | 27.89 | 0.07 | 0 | 18.63 | 0.04 | 0 | 16.94 | 0.03 |
|
|
| CodeLLaMA-13B | 0 | 4.42 | 0.04 | 0 | 4.89 | 0.13 | 0 | 5.10 | 0.04 | 0 | 4.71 | 0.08 | 22.42 | 25.60 | 0.06 | 0 | 13.67 | 0.21 | 0 | 6.90 | 0.33 |
|
|
| QWen2.5-Coder-14B | 0.42 | 1.75 | 0.25 | 0 | 0.68 | 0.34 | 0 | 0.90 | 0.37 | 0 | 1.10 | 0.45 | 13.45 | 27.41 | 0.19 | 0 | 15.01 | 0.26 | 0 | 12.59 | 0.11 |
|
|
| QWen2.5-Coder-32B | 0 | 0.88 | 0.23 | 0 | 0.57 | 0.19 | 0 | 1.19 | 0.44 | 0 | 1.22 | 0.32 | 15.02 | 26.55 | 0.21 | 0 | 14.63 | 0.27 | 0 | 12.14 | 0.09 |
|
|
| CodeLLaMA-34B | 0 | 9.76 | 0.21 | 0 | 12.01 | 0.41 | 0.13 | 13.07 | 0.19 | 0.69 | 20.53 | 0.35 | 16.98 | 25.93 | 0.11 | 0 | 18.62 | 0.44 | 0 | 9.86 | 0.28 |
|
|
|**BePilot-1.5B** |**57.55** |**77.98** | **0.87** |**74.03** |**88.45** | **0.94** | **60.11** | **74.24** | **0.76** | **58.70** | **84.81** | **0.88** |**40.09** |**48.77** | **0.50** |**29.10** |**71.62** |**0.74** |**29.31** |**78.77** |**0.80** |
|
|
|**BePilot-7B** |**69.26** |**84.78** | **0.88** |**78.07** |**90.62** | **0.94** | **68.82** | **80.45** | **0.81** | **68.58** | **88.35** | **0.89** |**51.97** |**59.50** | **0.60** |**35.77** |**73.21** |**0.77** |**35.46** |**80.98** |**0.83** |
|
|
|
|
|
|
- LLVM
|
|
|
|
| | Stmt.Comp. | Stmt.Comp. | Stmt.Comp. | Stmt.Comp. | Stmt.Comp. | Stmt.Comp. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Code. Gen. | Code. Gen. | Code. Gen. | Pro. Rep. | Pro. Rep. | Pro. Rep. | Pro. Rep. | Pro. Rep. | Pro. Rep. |
|
|
|--------------------- |------------ |------------ |------------ |------------ |------------ |------------ |------------- |------------- |------------- |------------- |------------- |------------- |:----------: |:----------: |:----------: |------------ |------------ |----------- |----------- |------------ |----------- |
|
|
| | C++ | C++ | C++ | TD | TD | TD | C++ | C++ | C++ | TD | TD | TD | C++ | C++ | C++ | C++ | C++ | C++ | TD | TD | TD |
|
|
| Model | EM | ED | R-L | EM | ED | R-L | EM | ED | R-L | EM | ED | R-L | CB | ED | R-L | EM | ED | R-L | EM | ED | R-L |
|
|
| DeepSeekCoder-1.3B | 0 | 5.03 | 0 | 0 | 8.48 | 0.01 | 0 | 7.45 | 0.01 | 0 | 7.99 | 0.01 | 22.36 | 25.22 | 0.06 | 0 | 14.47 | 0.05 | 0 | 13.23 | 0.06 |
|
|
| YiCoder-1.5B | 0 | 1.56 | 0.01 | 0 | 6.09 | 0.03 | 0 | 2.93 | 0.01 | 0 | 5.21 | 0.03 | 23.69 | 6.59 | 0.01 | 0 | 2.08 | 0.01 | 0 | 0.94 | 0.01 |
|
|
| QWen2.5-Coder-1.5B | 0 | 3.59 | 0.02 | 0 | 7.52 | 0.05 | 0.02 | 4.57 | 0.01 | 0 | 5.50 | 0.03 | 27.17 | 18.96 | 0.04 | 0 | 7.79 | 0.02 | 0 | 11.33 | 0.05 |
|
|
| CodeLLaMA-7B | 0 | 4.42 | 0.09 | 0 | 8.93 | 0.11 | 0.04 | 5.60 | 0.03 | 0.29 | 9.80 | 0.09 | 19.29 | 17.44 | 0.05 | 0 | 10.47 | 0.10 | 0 | 8.90 | 0.14 |
|
|
| CodeGemma-7B | 0 | 5.02 | 0.02 | 0 | 7.93 | 0.03 | 0 | 10.72 | 0.02 | 0 | 11.20 | 0.01 | 29.37 | 25.35 | 0.08 | 0 | 20.18 | 0.08 | 0 | 18.79 | 0.11 |
|
|
| QWen2.5-Coder-7B | 0 | 8.75 | 0.02 | 0 | 17.84 | 0.06 | 0 | 19.16 | 0.07 | 0 | 27.45 | 0.20 | 25.20 | 28.80 | 0.08 | 0 | 20.11 | 0.05 | 0 | 22.68 | 0.08 |
|
|
| CodeLLaMA-13B | 0 | 2.98 | 0.02 | 0 | 6.40 | 0.12 | 0.02 | 5.30 | 0.02 | 0 | 12.71 | 0.31 | 19.05 | 18.00 | 0.08 | 0 | 16.17 | 0.29 | 0 | 3.06 | 0.06 |
|
|
| QWen2.5-Coder-14B | 0.05 | 1.39 | 0.23 | 0 | 2.39 | 0.15 | 0 | 0.98 | 0.36 | 0 | 1.18 | 0.52 | 13.21 | 27.91 | 0.17 | 0 | 15.78 | 0.25 | 0 | 18.71 | 0.26 |
|
|
| QWen2.5-Coder-32B | 0 | 1.01 | 0.18 | 0 | 2.08 | 0.12 | 0 | 1.28 | 0.42 | 0 | 1.68 | 0.55 | 16.25 | 27.47 | 0.20 | 0 | 15.74 | 0.26 | 0 | 19.77 | 0.25 |
|
|
| CodeLLaMA-34B | 0.36 | 10.31 | 0.25 | 0 | 26.50 | 0.37 | 0.30 | 15.50 | 0.19 | 1.01 | 22.91 | 0.55 | 21.30 | 19.84 | 0.07 | 0 | 18.20 | 0.36 | 0 | 6.10 | 0.14 |
|
|
|**BePilot-1.5B** |**62.11** |**80.79** | **0.90** |**60.46** |**83.81** | **0.89** | **57.39** | **73.16** | **0.75** | **51.30** | **70.60** | **0.76** |**41.12** |**49.25** | **0.49** |**26.64** |**71.63** |**0.76** |**7.02** |**74.44** |**0.74** |
|
|
|**BePilot-7B** |**73.90** |**87.53** | **0.91** |**68.09** |**86.89** | **0.90** | **64.69** | **78.34** | **0.79** | **57.51** | **75.97** | **0.81** |**47.08** |**55.18** | **0.57** |**33.19** |**71.68** |**0.78** |**7.23** |**70.72** |**0.73** | |
