|
--- |
|
language: |
|
- en |
|
pipeline_tag: text-generation |
|
tags: |
|
- facebook |
|
- meta |
|
- pytorch |
|
- llama |
|
- llama-3 |
|
license: other |
|
license_name: llama3 |
|
license_link: LICENSE |
|
datasets: |
|
- JeanKaddour/minipile |
|
- raincandy-u/SlimOrca-Llama-3-Preference-DPO-Pairs |
|
--- |
|
|
|
|
|
|
|
 |
|
|
|
# Llama-3-5B-Sheard |
|
|
|
Pruned version of Llama-3-8b. |
|
|
|
Tool used: [PrunMe](https://github.com/arcee-ai/PruneMe), Mergekit. |
|
|
|
**Meta Llama 3 is licensed under the Meta Llama 3 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.** |
|
|
|
## Training |
|
|
|
After sliced by mergekit, the model is continue-pretrained on minipile for 1 epoch and ~100k samples. Then we trained it using ORPO on Llama-3-70b generated DPO pairs. |
|
|
|
## Disclaimer |
|
|
|
This model is for testing purposes only, and when the system prompt is not empty, the output may repeat and not stop! |
|
|
|
<h1><a href="https://discord.gg/Dw8SkfBm">Join our discord</a></h1> |
