--- tags: - code - starcoder2 library_name: transformers pipeline_tag: text-generation license: bigcode-openrail-m --- [![QuantFactory Banner](https://lh7-rt.googleusercontent.com/docsz/AD_4nXeiuCm7c8lEwEJuRey9kiVZsRn2W-b4pWlu3-X534V3YmVuVc2ZL-NXg2RkzSOOS2JXGHutDuyyNAUtdJI65jGTo8jT9Y99tMi4H4MqL44Uc5QKG77B0d6-JfIkZHFaUA71-RtjyYZWVIhqsNZcx8-OMaA?key=xt3VSDoCbmTY7o-cwwOFwQ)](https://hf.co/QuantFactory) # QuantFactory/starcoder2-3b-instruct-GGUF This is quantized version of [TechxGenus/starcoder2-3b-instruct](https://huggingface.co/TechxGenus/starcoder2-3b-instruct) created using llama.cpp # Original Model Card

starcoder2-instruct

### starcoder2-instruct We've fine-tuned starcoder2-3b with an additional 0.7 billion high-quality, code-related tokens for 3 epochs. We used DeepSpeed ZeRO 3 and Flash Attention 2 to accelerate the training process. It achieves **65.9 pass@1** on HumanEval-Python. This model operates using the Alpaca instruction format (excluding the system prompt). ### Usage Here give some examples of how to use our model: ```python from transformers import AutoTokenizer, AutoModelForCausalLM import torch PROMPT = """### Instruction {instruction} ### Response """ instruction = prompt = PROMPT.format(instruction=instruction) tokenizer = AutoTokenizer.from_pretrained("TechxGenus/starcoder2-3b-instruct") model = AutoModelForCausalLM.from_pretrained( "TechxGenus/starcoder2-3b-instruct", torch_dtype=torch.bfloat16, device_map="auto", ) inputs = tokenizer.encode(prompt, return_tensors="pt") outputs = model.generate(input_ids=inputs.to(model.device), max_new_tokens=2048) print(tokenizer.decode(outputs[0])) ``` With text-generation pipeline: ```python from transformers import pipeline import torch PROMPT = """### Instruction {instruction} ### Response """ instruction = prompt = PROMPT.format(instruction=instruction) generator = pipeline( model="TechxGenus/starcoder2-3b-instruct", task="text-generation", torch_dtype=torch.bfloat16, device_map="auto", ) result = generator(prompt, max_length=2048) print(result[0]["generated_text"]) ``` ### Note Model may sometimes make errors, produce misleading contents, or struggle to manage tasks that are not related to coding. It has undergone very limited testing. Additional safety testing should be performed before any real-world deployments.