|
--- |
|
license: mit |
|
license_link: https://huggingface.co/microsoft/phi-2/resolve/main/LICENSE |
|
language: |
|
- en |
|
pipeline_tag: text-generation |
|
tags: |
|
- nlp |
|
- code |
|
--- |
|
# MobiLlama-05B |
|
|
|
<center><img src="MobileLLaMa.png" alt="mobillama logo" width="300"/></center> |
|
|
|
## Model Summary |
|
|
|
MobiLlama-05B is a Small Language Model with **0.5 billion** parameters. It was trained using the Amber data sources [Amber-Dataset](https://huggingface.co/datasets/LLM360/AmberDatasets). |
|
|
|
|
|
## Model Description |
|
|
|
- **Model type:** Small Language Model (SLM) built using the architecture design of LLaMA-7B |
|
- **Language(s) (NLP):** English |
|
- **License:** Apache 2.0 |
|
- **Resources for more information:** |
|
- [Training Code](https://github.com/LLM360/amber-train) |
|
- [Data Preparation](https://github.com/LLM360/amber-data-prep) |
|
- [Metrics](https://github.com/LLM360/Analysis360) |
|
- [Fully processed Amber pretraining data](https://huggingface.co/datasets/LLM360/AmberDatasets) |
|
|
|
|
|
## How to Use |
|
|
|
MobiLlama-05B has been integrated in the development version (4.37.0.dev) of `transformers`. Until the official version is released through `pip`, ensure that you are doing one of the following: |
|
|
|
* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function. |
|
|
|
* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source. |
|
|
|
The current `transformers` version can be verified with: `pip list | grep transformers`. |
|
|
|
To load a specific checkpoint, simply pass a revision with a value between `"ckpt_000"` and `"ckpt_358"`. If no revision is provided, it will load `"ckpt_359"`, which is the final checkpoint. |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
model = AutoModelForCausalLM.from_pretrained("MBZUAI/MobiLlama-05B", torch_dtype="auto", trust_remote_code=True) |
|
tokenizer = AutoTokenizer.from_pretrained("MBZUAI/MobiLlama-05B", trust_remote_code=True) |
|
|
|
text = "I was dancing in the river when " |
|
input_ids = tokenizer(text, return_tensors="pt").to('cuda').input_ids |
|
outputs = model.generate(input_ids, max_length=1000, repetition_penalty=1.2, pad_token_id=tokenizer.eos_token_id) |
|
print(tokenizer.batch_decode(outputs[:, input_ids.shape[1]:-1])[0].strip()) |
|
|
|
``` |
|
|
|
## Evaluation |
|
| Evaluation Benchmark | MobiLlama-0.5B | MobiLlama-0.8B | MobiLlama-1.2B | |
|
| ----------- | ----------- | ----------- | |
|
| HellaSwag | 0.5252 | 0.5409 | 0.6299 | |
|
| MMLU | 0.2645 | 0.2692 | 0.2423 | |
|
| Arc Challenge | 0.2952 | 0.3020 | 0.3455 | |
|
| TruthfulQA | 0.3805 | 0.3848 | 0.3557 | |
|
| CrowsPairs | 0.6403 | 0.6482 | 0.6812 | |
|
| PIQA | 0.7203 | 0.7317 | 0.7529 | |
|
| Race | 0.3368 | 0.3337 | 0.3531 | |
|
| SIQA | 0.4022 | 0.4160 | 0.4196 | |
|
| Winogrande | 0.5753 | 0.5745 | 0.6108 | |
|
|
|
## Intended Uses |
|
|
|
Given the nature of the training data, the MobiLlama-05B model is best suited for prompts using the QA format, the chat format, and the code format. |
|
|
|
|
