File size: 2,096 Bytes
dc703c3 50c23f7 dc703c3 50c23f7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 |
---
license: cc-by-sa-4.0
datasets:
- bigcode/the-stack-dedup
- sahil2801/CodeAlpaca-20k
- teknium/GPTeacher-CodeInstruct
model-base:
- replit/replit-code-v1-3b
tags:
- code
- instruct
- self instruct
language:
- code
programming_language:
- Markdown
- Java
- JavaScript
- Python
- TypeScript
- PHP
- SQL
- JSX
- reStructuredText
- Rust
- C
- CSS
- Go
- C++
- HTML
- Vue
- Ruby
- Jupyter Notebook
- R
- Shell
---
Base Model: replit/replit-code-v1-3b
This model is fine tuned on both Sahil2801's CodeAlpaca & Teknium's GPTeacher Code-Instruct to give Replit's Code model instruct capabilities.
Dataset links:
CodeAlpaca: https://huggingface.co/datasets/sahil2801/CodeAlpaca-20k
GPTeacher subset - Code Instruct: https://github.com/teknium1/GPTeacher
This model was trained on 2x a100 80gb for 1 hour on ~25,000 code instruction/response pairs in Alpaca format.
Refer to the base models HuggingFace model card for some basic requirements to run: https://huggingface.co/replit/replit-code-v1-3b
This fine tune can be prompted like any alpaca fine tune:
```
### Instruction:
<prompt>
### Input:
<additional context>
### Response:
```
or
```
### Instruction:
<prompt>
### Response:
```
This model for me produced coherent outputs with the following sampler settings, but feel free to experiment:
```
max_new_tokens=128, do_sample=True, use_cache=True, temperature=0.2, top_p=0.9, eos_token_id= self.tokenizer.eos_token_id
```
In the tokenizer decode arguments, it also needs these settings:
```
skip_special_tokens=True, clean_up_tokenization_space=False
```
The following parameters were used with HuggingFace trainer to train the model with:
```
--model_name_or_path replit/replit-code-v1-3b --data_path /root/stanford_alpaca/train.json --bf16 True --output_dir /root/stanford_alpaca/model_ckpts --num_train_epochs 3 --per_device_train_batch_size 4 --per_device_eval_batch_size 1 --gradient_accumulation_steps 8 --save_strategy steps --save_steps 200 --save_total_limit 3 --learning_rate 1e-5 --weight_decay 0. --warmup_ratio 0.03 --tf32 True --run_name Replit1
```
|