|
--- |
|
license: cc-by-nc-sa-4.0 |
|
pipeline_tag: text-generation |
|
language: |
|
- en |
|
tags: |
|
- finetuned |
|
--- |
|
|
|
# Model Card for ZoyLLM-7B-SlimOrca |
|
|
|
The ZoyLLM-7B-SlimOrca Large Language Model (LLM) is a LoRA-finetuned generative text model with Mistral-7B-v0.1 is the base model. |
|
Mistral-7B-v0.1 outperforms Llama 2 13B on all benchmarks we tested. |
|
|
|
## Model Architecture |
|
|
|
ZoyLLM-7B-SlimOrca is a transformer model, with the following architecture choices: |
|
- Grouped-Query Attention |
|
- Sliding-Window Attention |
|
- Byte-fallback BPE tokenizer |
|
|
|
## Datasets |
|
- Self-introduction (20 samples) |
|
- SlimOrca (100k samples random sampled) |
|
- EverythingLM v3 |
|
|
|
## Template |
|
We finetuned the model using a template similar to the dolphin chat template |
|
``` |
|
<|im_start|>system |
|
{system}<|im_end|> |
|
<|im_start|>user |
|
{prompt}<|im_end|> |
|
<|im_start|>assistant |
|
``` |
|
|
|
## Troubleshooting |
|
|
|
- If you see the following error: |
|
``` |
|
KeyError: 'mistral' |
|
``` |
|
- Or: |
|
``` |
|
NotImplementedError: Cannot copy out of meta tensor; no data! |
|
``` |
|
|
|
Ensure you are utilizing a stable version of Transformers, 4.34.0 or newer. |
|
|
|
## The Zoy AI Team |
|
|
|
Pham Tung Lam, Nguyen Duc Nhan. |
