This model is part of LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation.
LoRI (LoRA with Reduced Interference) is a simple yet effective parameter-efficient fine-tuning (PEFT) method designed for Large Language Models (LLMs) to mitigate notable overhead and address parameter interference in multi-task scenarios. It achieves this by freezing the projection matrices A as random projections and sparsifying the matrices B using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance. Moreover, LoRI minimizes cross-task interference in adapter merging by leveraging the orthogonality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting.
LoRI-D_nlu_llama3_rank_32 is a LoRI adapter specifically fine-tuned for Natural Language Understanding (NLU) tasks. It is based on the meta-llama/Meta-Llama-3-8B base model with a LoRA rank of 32. This model leverages LoRI's design to offer efficient fine-tuning, reduced parameter overhead, and minimized cross-task interference.
meta-llama/Meta-Llama-3-8BThis model is intended for use in fine-tuning Large Language Models for various tasks, including:
It is particularly useful for researchers and practitioners looking for parameter-efficient fine-tuning solutions that reduce cross-task interference in multi-task and continual learning settings.
This model is not intended for:
As an adapter fine-tuned on a base Large Language Model (e.g., Llama-3-8B), this model inherits potential biases present in its foundational training data. This could lead to biased, harmful, or undesirable outputs. While LoRI aims to reduce cross-task interference, complex interactions in highly diverse multi-task setups might still occur. Performance on out-of-distribution data or tasks not covered during its training may vary.
Users (both direct and downstream) should be made aware of the risks, biases, and limitations of the model. It is recommended to perform thorough evaluations specific to your application and data before deployment. Implement additional filtering or human-in-the-loop validation for critical use cases.
Pretrained LoRI adapters are available on the Hugging Face Hub. To use this specific NLU adapter with its base model (meta-llama/Meta-Llama-3-8B), follow the example below:
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
# Load the base model (Meta-Llama-3-8B)
base_model_name = "meta-llama/Meta-Llama-3-8B"
base_model = AutoModelForCausalLM.from_pretrained(
base_model_name,
torch_dtype=torch.bfloat16, # Use bfloat16 for Llama-3 if your hardware supports it
device_map="auto" # Automatically maps model to available devices (e.g., GPU)
)
# Load the LoRI-D NLU adapter on top of the base model
lori_adapter_name = "tomg-group-umd/LoRI-D_nlu_llama3_rank_32"
model = PeftModel.from_pretrained(base_model, lori_adapter_name)
# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
# Example for text generation
prompt = "The quick brown fox jumps over the lazy"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device) # Move inputs to model's device
# Generate text
model.eval() # Set model to evaluation mode
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=50, temperature=0.7, do_sample=True)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Prompt: {prompt}")
print(f"Generated: {generated_text}")
LoRI models are trained and evaluated on a variety of datasets covering different tasks:
For more detailed information on specific datasets used for each task, please refer to the LoRI GitHub repository and the accompanying paper.
LoRI is implemented using Fully Sharded Data Parallel (FSDP) and can be executed in a multi-GPU environment. The training process typically involves two stages:
The provided training scripts in the GitHub repository support LLaMA-3-8B and Mistral-7B base models with adapter ranks of 32 and 64, performing both LoRI-D and LoRI-S training, followed by evaluation on downstream tasks.
The specific LoRA parameters for this LoRI-D_nlu_llama3_rank_32 adapter, as extracted from adapter_config.json, are:
r: 32lora_alpha: 64lora_dropout: 0.05target_modules: ["down_proj", "up_proj", "k_proj", "gate_proj", "v_proj", "q_proj", "o_proj"]peft_type: LORAThe model's performance was evaluated extensively across various benchmarks relevant to Natural Language Understanding, Code Generation, Mathematical Reasoning, and Safety Alignment.
Performance was evaluated on standard benchmarks for each task. Specific datasets include HumanEval for code generation, GSM8K for mathematical reasoning, and Saferpaca for safety alignment.
Evaluations disaggregated by tasks (NLU, code, math, safety) and potentially by model size/rank.
Performance was measured using standard metrics relevant to each task (e.g., accuracy for NLU/math, pass@1 for code generation).
Extensive experiments demonstrated that LoRI consistently outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than standard LoRA. In multi-task experiments, LoRI enabled effective adapter merging and continual learning with significantly reduced cross-task interference. For detailed evaluation results and comparisons, please refer to the LoRI paper.
LoRI introduces modifications to the standard LoRA architecture. It freezes the low-rank projection matrix A as random projections and sparsifies the B matrix using task-specific masks. This approach aims to achieve mutual orthogonality between adapter subspaces, reduce cross-task interference, and significantly decrease the number of trainable parameters.
Training and evaluation were conducted on GPUs. Specific details on hardware configurations might be found within the supplementary materials of the paper or the GitHub repository's training scripts.
The codebase primarily leverages PyTorch for deep learning, along with the transformers and peft libraries from Hugging Face.
If you use LoRI in your work, please cite the following paper:
@article{zhang2025lori,
title={LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation},
author={Zhang, Juzheng and You, Jiacheng and Panda, Ashwinee and Goldstein, Tom},
journal={arXiv preprint arXiv:2504.07448},
year={2025}
}
APA: [More Information Needed]
Niels, Hugging Face Community Science team.
For questions about the model or LoRI project, please contact [email protected].
Base model
meta-llama/Meta-Llama-3-8B