metadata

license: mit
license_link: https://huggingface.co/microsoft/phi-2/resolve/main/LICENSE
language:
  - en
pipeline_tag: text-generation
tags:
  - nlp
  - code

MobiLlama-05B

Model Summary

MobiLlama-05B is a Small Language Model with 0.5 billion parameters. It was trained using the Amber data sources Amber-Dataset.

Model Description

Model type: Small Language Model (SLM) built using the architecture design of LLaMA-7B
Language(s) (NLP): English
License: Apache 2.0
Resources for more information:

How to Use

MobiLlama-05B has been integrated in the development version (4.37.0.dev) of transformers. Until the official version is released through pip, ensure that you are doing one of the following:

When loading the model, ensure that trust_remote_code=True is passed as an argument of the from_pretrained() function.
Update your local transformers to the development version: pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers. The previous command is an alternative to cloning and installing from the source.

The current transformers version can be verified with: pip list | grep transformers.

To load a specific checkpoint, simply pass a revision with a value between "ckpt_000" and "ckpt_358". If no revision is provided, it will load "ckpt_359", which is the final checkpoint.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("MBZUAI/MobiLlama-05B", torch_dtype="auto", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("MBZUAI/MobiLlama-05B", trust_remote_code=True)

text = "I was dancing in the river when "
input_ids = tokenizer(text, return_tensors="pt").to('cuda').input_ids
outputs = model.generate(input_ids, max_length=1000, repetition_penalty=1.2, pad_token_id=tokenizer.eos_token_id)
print(tokenizer.batch_decode(outputs[:, input_ids.shape[1]:-1])[0].strip())

Evaluation

| Evaluation Benchmark | MobiLlama-0.5B | MobiLlama-0.8B | MobiLlama-1.2B | | ----------- | ----------- | ----------- | | HellaSwag | 0.5252 | 0.5409 | 0.6299 | | MMLU | 0.2645 | 0.2692 | 0.2423 | | Arc Challenge | 0.2952 | 0.3020 | 0.3455 | | TruthfulQA | 0.3805 | 0.3848 | 0.3557 | | CrowsPairs | 0.6403 | 0.6482 | 0.6812 | | PIQA | 0.7203 | 0.7317 | 0.7529 | | Race | 0.3368 | 0.3337 | 0.3531 | | SIQA | 0.4022 | 0.4160 | 0.4196 | | Winogrande | 0.5753 | 0.5745 | 0.6108 |

Intended Uses

Given the nature of the training data, the MobiLlama-05B model is best suited for prompts using the QA format, the chat format, and the code format.