File size: 4,457 Bytes
b243ee5 a946264 7519a87 1e9d85d d77260d 7519a87 d77260d 9bb8967 d77260d 5eac73c 1e9d85d 5eac73c b9967d2 7519a87 5eac73c 7519a87 5eac73c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 |
---
datasets:
- fbnhnsl/FIM_Solidity_Dataset
language:
- en
metrics:
- bleu
- meteor
base_model:
- deepseek-ai/deepseek-coder-1.3b-base
pipeline_tag: text-generation
tags:
- code
license: mit
---
This is a fine-tuned deepseek-coder-1.3b-base model for automatic completion of Solidity code. The model was fine-tuned using the Parameter Efficient Fine-tuning (PEFT) method
Quantized Low Rank Adaptation (QLoRA) and a Fill-in-the-Middle (FIM) transformed dataset consisting of Solidity constructs (functions, modifiers, mappings, etc.). The model has a maximum sequence length of 256 tokens.
General Fine-tuning informations:
- Epochs: 2
- Optimizer: paged AdamW 8-bit
- Batch size: 8
- LoRA target modules: ["q_proj", "o_proj", "k_proj", "v_proj"]
- Quantization type: normal float 4-bit
- QLoRA compute type: brain float 16-bit
- Total time: 1 hour 23 minutes
- Accelerator: Nvidia L4 Tensor Core GPU
Some of the Hyperparameters were determined using Hyperparameter optimization with Ray Tune. The corresponding result for the best trial were:
- Learning rate: 0.00016
- Weight decay: 0.0534
- Warmup steps: 100
- Gradient accumulation steps: 2
- LoRA rank: 64
- LoRA alpha: 64
- LoRA dropout: 0.0934665
The Fine-tuning results are:
- Training loss: ~0.7
- Validation loss: ~0.75
The model was evaluated with the test split compared to the base model. The metrics were used: Perplexity, BLEU and METEOR. The Perplexity results are:
- Perplexity Base Model: 12.08
- Perplexity Fine-tuned Model: 2.19
The following code shows an example of how to use the model:
```python
# Load the fine-tuned model
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
pretrained_checkpoint = 'deepseek-ai/deepseek-coder-1.3b-base'
finetuned_checkpoint = 'path/to/model'
tokenizer = AutoTokenizer.from_pretrained(finetuned_checkpoint)
old_model = AutoModelForCausalLM.from_pretrained(pretrained_checkpoint)
old_model.resize_token_embeddings(len(tokenizer))
finetuned_model = PeftModel.from_pretrained(old_model, checkpoint).to(device)
# ----------------------------------------------------------------------------
# General automatic code completion
code_example = '''<|secure_function|>\tfunction add('''
model_inputs = tokenizer(code_example, return_tensors="pt").to(device)
input_ids = model_inputs["input_ids"]
attention_mask = model_inputs["attention_mask"]
generated_ids = finetuned_model.generate(input_ids,
do_sample=True,
max_length=256,
num_beams=4,
temperature=0.3,
pad_token_id=tokenizer.eos_token_id,
attention_mask=attention_mask)
print(tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0])
# Expected output:
# function add(uint256 a, uint256 b) internal pure returns (uint256) {
# return a + b;
# }
# ----------------------------------------------------------------------------
# Fill-in-the-middle
def generate_fim(prefix, suffix, model, tokenizer, max_length=256):
input_text = f"<|fim_begin|>{prefix}<|fim_hole|>{suffix}<|fim_end|>"
inputs = tokenizer.encode(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(
inputs,
max_length=max_length,
num_beams=8,
temperature=0.3,
do_sample=True,
pad_token_id=tokenizer.eos_token_id
)
middle = tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True)
return prefix + middle + suffix
prefix = '''pragma solidity ^0.8.0;\n\n'''
suffix = '''\n\ncontract FOO is Context, IERC20, Ownable {'''
print(generate_fim(prefix, suffix, finetuned_model, tokenizer))
# Expected output:
# pragma solidity ^0.8.0;
#
# import "@openzeppelin/contracts/utils/Context.sol" as Context;
# import "@openzeppelin/contracts/interfaces/IERC20.sol" as IERC20;
# import "@openzeppelin/contracts/access/Ownable.sol" as Ownable;
#
# contract FOO is Context, IERC20, Ownable {
```
If you wish to use this model, you can cite it as follows:
```latex
@misc{hensel2025fim_model,
title = {Finetuned deepseek-coder-1.3b-base model for automatic code completion of Solidity code},
author={Fabian Hensel},
year={2025}
}
``` |