metadata
license: cc-by-nc-sa-4.0
pipeline_tag: text-generation
language:
- en
tags:
- finetuned
Model Card for ZoyLLM-7B-SlimOrca
The ZoyLLM-7B-SlimOrca Large Language Model (LLM) is a LoRA-finetuned generative text model with Mistral-7B-v0.1 is the base model. Mistral-7B-v0.1 outperforms Llama 2 13B on all benchmarks we tested.
Model Architecture
ZoyLLM-7B-SlimOrca is a transformer model, with the following architecture choices:
- Grouped-Query Attention
- Sliding-Window Attention
- Byte-fallback BPE tokenizer
Datasets
- Self-introduction (20 samples)
- SlimOrca (100k samples random sampled)
- EverythingLM v3
Template
We finetuned the model using a template similar to the dolphin chat template
<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
Troubleshooting
- If you see the following error:
KeyError: 'mistral'
- Or:
NotImplementedError: Cannot copy out of meta tensor; no data!
Ensure you are utilizing a stable version of Transformers, 4.34.0 or newer.
The Zoy AI Team
Pham Tung Lam, Nguyen Duc Nhan.