LORA-Flow

LoRAs and fusion gates for our paper

We released all of our checkpoints used in LoRA-Flow which has been accepted to ACL 2024 main conference.

Summary

In this repo, we release LoRA modules and the gate of 7B models trained in our paper in HuggingFace format.

Introduction

LoRA-Flow provides an efficient way to fuse different LoRA modules. The following picture shows our proposed method, we use layer-wise fusion gates to facilitate dynamic LoRA fusion, which project input hidden states of each layer into fusion weights. LoRA-flow can be applied into Llama-7b backbone . For more details, please refer to our paper.

Training Data

Data used for LoRA modules

For the language LoRA modules: we use the 52K training samples from Okapi for each language, respectively.

For the math LoRA module: we use Metamath that is comprised of 395K mathematical problems and the corresponding solutions in English.

For the code LoRA module: we use the Magicoder dataset Magicoder, which consists of 186K code generation problems and the corresponding solutions in English.

Data used for gates

We use gates to fuse different LoRA modules. We employ few-shot training and have released our training data. For more details, please refer to our GitHub.

Results

We have released the results for LoRAs and LoRA-Flow

Method	MGSM (Math)				HumanEval (Code)
Method	Zh	Ru	Es	Avg.	Zh	Ru	Es	Avg.
Base Model	4.4	3.2	2.4	3.3	0.0	0.0	2.4	0.8
Single LoRA	5.2	3.6	3.6	4.1	12.2	14.0	10.4	12.2
Single LoRA	26.8	32.8	41.2	33.6	18.3	23.2	21.9	21.1
LoRA Fusion	12.8	10.4	18.4	13.9	17.1	17.7	18.3	17.7
	20.8	28.4	36.8	28.7	19.5	21.3	20.1	20.3
	33.2	37.6	42.0	37.6	20.7	23.8	23.2	22.6

Citation

@inproceedings{wang-etal-2024-lora-flow,
    title = "LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks",
    author = "Wang, Hanqing  and
      Ping, Bowen  and
      Wang, Shuo  and
      Han, Xu  and
      Chen, Yun  and
      Liu, Zhiyuan  and
      Sun, Maosong",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.695",
    doi = "10.18653/v1/2024.acl-long.695",
    pages = "12871--12882"
}