|
--- |
|
language: en |
|
tags: |
|
- llama |
|
- text-generation |
|
- model-merging |
|
license: mit |
|
base_model: |
|
- meta-llama/Meta-Llama-3-8B |
|
library_name: transformers |
|
--- |
|
|
|
# llama-3-8b-merged-linear |
|
|
|
## Overview |
|
This model represents a linear merge of three distinct Llama 3-8b models using the Mergekit tool. The primary goal of this merge is to leverage the unique strengths of each base model, such as multilingual capabilities and specialized domain knowledge, into a more versatile and generalized language model. |
|
|
|
By merging these models linearly, we combine their expertise into a unified model that performs well across various tasks, such as text generation, multilingual understanding, and domain-specific tasks. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
- **Models Used**: |
|
- Danielbrdz/Barcenas-Llama3-8b-ORPO |
|
- DeepMount00/Llama-3-8b-Ita |
|
- lightblue/suzume-llama-3-8B-multilingual |
|
|
|
- **Merging Tool**: Mergekit |
|
- **Merge Method**: Linear merge with equal weighting (1.0) for all models |
|
- **Tokenizer Source**: Union |
|
- **Data Type**: float16 (FP16) precision |
|
- **License**: MIT License |
|
- **Languages Supported**: Multilingual, including English, Italian, and potentially others from the multilingual base models |
|
|
|
## Configuration |
|
The following YAML configuration was used to produce this model: |
|
|
|
```yaml |
|
models: |
|
- model: Danielbrdz/Barcenas-Llama3-8b-ORPO |
|
parameters: |
|
weight: 1.0 |
|
- model: DeepMount00/Llama-3-8b-Ita |
|
parameters: |
|
weight: 1.0 |
|
- model: lightblue/suzume-llama-3-8B-multilingual |
|
parameters: |
|
weight: 1.0 |
|
merge_method: linear |
|
tokenizer_source: union |
|
dtype: float16 |
