Llama 3.1 Merged Adapters

Model Description

This is a merged model combining multiple fine-tuned LoRA adapters using TIES (Task Inference with Expert Selection) merging technique. The model combines the strengths of multiple specialized adapters into a single model.

Base Model

unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit

Merged Adapters

The following adapters were merged:

llama337 - Specialized for creative writing
llama338 - Specialized for logical reasoning
llama340 - Specialized for code generation
llama344 - Specialized for instruction following
llama345 - Specialized for factual knowledge
llama346 - Specialized for conversational abilities
llama349 - Specialized for problem solving
llama350 - Specialized for structured output

Merging Parameters

Merging Method: TIES (Task Inference with Expert Selection)
Density: 0.2 (controls parameter sparsity)
Weights: Equal weighting (1.0 for each adapter)
Merge Date: 2025-03-09

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the model and tokenizer
model = AutoModelForCausalLM.from_pretrained("kevin009/llama3-merged-adapters")
tokenizer = AutoTokenizer.from_pretrained("kevin009/llama3-merged-adapters")

# Example usage
prompt = "Write a short story about a robot learning to paint."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Model Performance

This merged model combines the capabilities of multiple specialized adapters, resulting in improved performance across a variety of tasks compared to individual adapters.

Limitations

The model inherits limitations from the base Llama 3.1 model
May produce inconsistent outputs for certain edge cases
As with all language models, can produce incorrect or misleading information

License

This model is subject to the license of the original Llama 3.1 model.