|
--- |
|
license: apache-2.0 |
|
language: |
|
- fr |
|
- it |
|
- de |
|
- es |
|
- en |
|
- zh |
|
inference: false |
|
--- |
|
# Model Card for Mobius-12B-base-m1 |
|
The Mobius-12B-base-m1 Large Language Model (LLM) is a pretrained model based on RWKV v5 arch. |
|
We utilized 0.01 billion tokens to conduct post-training on this model for alignment benchmarks, excluding the utilization of [DPO and SFT](https://github.com/BBuf/trl/pull/1). The process took approximately 10 hours, employing 4 * a800. |
|
|
|
|
|
## Warning |
|
This repo contains weights that are not compatible with Hugging Face [transformers](https://github.com/huggingface/transformers) library yet. But you can try this[PR](https://github.com/huggingface/transformers/pull/26963) as well. |
|
[RWKV runner]() or [AI00 server]() also work. |
|
|
|
## Instruction|Chat format |
|
|
|
This format must be strictly respected, otherwise the model will generate sub-optimal outputs. |
|
|
|
The template used to build a prompt for the Instruct model is defined as follows: |
|
``` |
|
User: {Instruction|prompt}\n\nAssistant: |
|
``` |
|
|
|
## Run the model |
|
|
|
[need to convert checkpoint to HF format](https://github.com/xiaol/RWKV-World-HF-Tokenizer?tab=readme-ov-file#huggingface-rwkv-world-model-convert) |
|
|
|
Need to install this [PR](https://github.com/huggingface/transformers/pull/26963) |
|
pip install -e git://github.com/BBuf/transformers.git |
|
|
|
```python |
|
import torch |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
model = AutoModelForCausalLM.from_pretrained("TimeMobius/Mobius-12B-base-m1", torch_dtype=torch.float16).to(0) |
|
tokenizer = AutoTokenizer.from_pretrained("TimeMobius/Mobius-12B-base-m1", trust_remote_code=True) |
|
|
|
text = "x" |
|
prompt = f'Question: {text.strip()}\n\nAnswer:' |
|
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(0) |
|
output = model.generate(inputs["input_ids"], max_new_tokens=40) |
|
print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True)) |
|
``` |
|
|
|
## Limitations |
|
|
|
The Mobius base m1 is the base model can be easily fine-tuned to achieve compelling performance. |
|
if you wanna better benchmark results use [DPO and SFT](https://github.com/BBuf/trl/pull/1) ,details in [readme](https://github.com/BBuf/trl/pull/1/files) |
|
|
|
### Benchmark |
|
|
|
| Mobius-12B-base-m1 | | |
|
|--------------------|----------| |
|
| lambda ppl | 3.41 | |
|
| lambda | 0.72 | |
|
| piqa | 0.78 | |
|
| hellaswag 10 shots | 0.72 | |
|
| winogrande | 0.68 | |
|
| arc_challenge 25shots | 0.47 | |
|
| arc_easy | 0.73 | |
|
| openbookqa | 0.40 | |
|
| sciq | 0.93 | |
|
|
|
|
|
# @TimeMobius |
