BLOOM (560M ckpt) fine-tuned on The Stack RUST code

Latest ckpt: https://huggingface.co/mrm8488/bloom-560m-finetuned-the-stack-rust/tree/100k

Model 🧠

BigScience Large Open-science Open-access Multilingual Language Model 🌸

BLOOM is an autoregressive Large Language Model (LLM), trained to continue text from a prompt on vast amounts of text data using industrial-scale computational resources. As such, it is able to output coherent text in 46 languages and 13 programming languages that is hardly distinguishable from text written by humans. BLOOM can also be instructed to perform text tasks it hasn't been explicitly trained for, by casting them as text generation tasks.

Dataset 📚

Rust 🦀 part of The Stack.

The Stack contains over 6TB of permissively-licensed source code files covering 358 programming languages. The dataset was created as part of the BigCode Project, an open scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs). The Stack serves as a pre-training dataset for Code LLMs, i.e., code-generating AI systems which enable the synthesis of programs from natural language descriptions as well as other from code snippets.

Example of usage 👩‍💻

import torch
from transformers import BloomTokenizerFast, BloomForCausalLM

device = 'cuda' if torch.cuda.is_available() else 'cpu'
ckpt = 'mrm8488/bloom-560m-finetuned-the-stack-rust'
revision = '100k' # latest one at the moment 

tokenizer = BloomTokenizerFast.from_pretrained(ckpt)
model = BloomForCausalLM.from_pretrained(ckpt, revision=revision).to(device)

def complete_code(text):
    inputs = tokenizer(text, return_tensors='pt')
    input_ids = inputs.input_ids.to(device)
    attention_mask = inputs.attention_mask.to(device)
    output = model.generate(input_ids, attention_mask=attention_mask, max_length=2048, eos_token_id=tokenizer.eos_token_id)

    return tokenizer.decode(output[0], skip_special_tokens=False)
    
code_prompt = """
use fastly::{Error, Request, Response};
use serde_json::{json, Value};

#[fastly::main]
fn main(req: Request) -> Result<Response, Error> {
  let mut response = req.send("origin_0")?;
"""

complete_code(code_prompt)

Citation ✒️

@misc {manuel_romero_2022,
    author       = { {Manuel Romero} },
    title        = { bloom-560m-finetuned-the-stack-rust (Revision 5358462) },
    year         = 2022,
    url          = { https://huggingface.co/mrm8488/bloom-560m-finetuned-the-stack-rust },
    doi          = { 10.57967/hf/0236 },
    publisher    = { Hugging Face }
}