File size: 1,117 Bytes
581e9ea 34aa2e5 581e9ea 34aa2e5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
---
license: cc-by-nc-sa-4.0
pipeline_tag: text-generation
language:
- en
tags:
- finetuned
---
# Model Card for ZoyLLM-7B-SlimOrca
The ZoyLLM-7B-SlimOrca Large Language Model (LLM) is a LoRA-finetuned generative text model with Mistral-7B-v0.1 is the base model.
Mistral-7B-v0.1 outperforms Llama 2 13B on all benchmarks we tested.
## Model Architecture
ZoyLLM-7B-SlimOrca is a transformer model, with the following architecture choices:
- Grouped-Query Attention
- Sliding-Window Attention
- Byte-fallback BPE tokenizer
## Datasets
- Self-introduction (20 samples)
- SlimOrca (100k samples random sampled)
- EverythingLM v3
## Template
We finetuned the model using a template similar to the dolphin chat template
```
<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```
## Troubleshooting
- If you see the following error:
```
KeyError: 'mistral'
```
- Or:
```
NotImplementedError: Cannot copy out of meta tensor; no data!
```
Ensure you are utilizing a stable version of Transformers, 4.34.0 or newer.
## The Zoy AI Team
Pham Tung Lam, Nguyen Duc Nhan. |