File size: 1,002 Bytes
b7c0990 3d58689 b7c0990 cfb643f b7c0990 3d58689 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
---
language:
- en
pipeline_tag: text-generation
tags:
- facebook
- meta
- pytorch
- llama
- llama-3
license: other
license_name: llama3
license_link: LICENSE
datasets:
- JeanKaddour/minipile
- raincandy-u/SlimOrca-Llama-3-Preference-DPO-Pairs
---
# Llama-3-5B-Sheard
Pruned version of Llama-3-8b.
Tool used: [PrunMe](https://github.com/arcee-ai/PruneMe), Mergekit.
**Meta Llama 3 is licensed under the Meta Llama 3 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.**
## Training
After sliced by mergekit, the model is continue-pretrained on minipile for 1 epoch and ~100k samples. Then we trained it using ORPO on Llama-3-70b generated DPO pairs.
## Disclaimer
This model is for testing purposes only, and when the system prompt is not empty, the output may repeat and not stop!
<h1><a href="https://discord.gg/Dw8SkfBm">Join our discord</a></h1> |