|
--- |
|
license: mit |
|
--- |
|
<div align="center"> |
|
|
|
<!-- <img src="title.png" alt="LoRA-Flow" width="200"> --> |
|
<!-- ***LORA-Flow*** --> |
|
<b><i style="font-size: 24px;">LORA-Flow</i></b> |
|
|
|
LoRAs and fusion gates for our paper |
|
|
|
<p align="center"> |
|
<a href="https://aclanthology.org/2024.acl-long.695/">Paper</a> • |
|
<a href="https://github.com/pingbowen23/LoRA-Flow"> Github</a> |
|
</p> |
|
</div> |
|
|
|
We released all of our checkpoints used in [LoRA-Flow](https://aclanthology.org/2024.acl-long.695.pdf) which has been accepted to ACL 2024 main conference. |
|
# Summary |
|
> In this repo, we release LoRA modules and the gate of 7B models trained in our paper in HuggingFace format. |
|
# Introduction |
|
LoRA-Flow provides an efficient way to fuse different LoRA modules. The following picture shows our proposed method, we use layer-wise fusion gates to facilitate dynamic LoRA fusion, which project input hidden states of each layer into fusion weights. LoRA-flow can be applied into [Llama-7b backbone](https://huggingface.co/meta-llama/Llama-2-7b) . For more details, please refer to our paper. |
|
 |
|
# Training Data |
|
## Data used for LoRA modules |
|
For the language LoRA modules: we use the 52K training samples from [Okapi](https://aclanthology.org/2023.emnlp-demo.28) for each language, respectively. |
|
|
|
For the math LoRA module: we use [Metamath](https://arxiv.org/abs/2309.12284) that is comprised of 395K mathematical problems and the corresponding solutions in English. |
|
|
|
For the code LoRA module: we use the Magicoder dataset [Magicoder](https://arxiv.org/abs/2312.02120), which consists of 186K code generation problems and the corresponding solutions in English. |
|
|
|
## Data used for gates |
|
We use gates to fuse different LoRA modules. We employ few-shot training and have released our training data. For more details, please refer to our GitHub. |
|
|
|
# Results |
|
We have released the results for LoRAs and LoRA-Flow |
|
|
|
|
|
<table border="1" cellspacing="0" cellpadding="10"> |
|
<thead> |
|
<tr> |
|
<th rowspan="2">Method</th> |
|
<th colspan="4">MGSM (Math)</th> |
|
<th colspan="4">HumanEval (Code)</th> |
|
</tr> |
|
<tr> |
|
<th style="text-align: right;">Zh</th> |
|
<th style="text-align: right;">Ru</th> |
|
<th style="text-align: right;">Es</th> |
|
<th style="text-align: right;">Avg.</th> |
|
<th style="text-align: right;">Zh</th> |
|
<th style="text-align: right;">Ru</th> |
|
<th style="text-align: right;">Es</th> |
|
<th style="text-align: right;">Avg.</th> |
|
</tr> |
|
</thead> |
|
<tbody> |
|
<tr> |
|
<td>Base Model</td> |
|
<td style="text-align: right;">4.4</td> |
|
<td style="text-align: right;">3.2</td> |
|
<td style="text-align: right;">2.4</td> |
|
<td style="text-align: right;">3.3</td> |
|
<td style="text-align: right;">0.0</td> |
|
<td style="text-align: right;">0.0</td> |
|
<td style="text-align: right;">2.4</td> |
|
<td style="text-align: right;">0.8</td> |
|
</tr> |
|
<tr> |
|
<td rowspan="2">Single LoRA</td> |
|
<td style="text-align: right;">5.2</td> |
|
<td style="text-align: right;">3.6</td> |
|
<td style="text-align: right;">3.6</td> |
|
<td style="text-align: right;">4.1</td> |
|
<td style="text-align: right;">12.2</td> |
|
<td style="text-align: right;">14.0</td> |
|
<td style="text-align: right;">10.4</td> |
|
<td style="text-align: right;">12.2</td> |
|
</tr> |
|
<tr> |
|
<td style="text-align: right;">26.8</td> |
|
<td style="text-align: right;">32.8</td> |
|
<td style="text-align: right;">41.2</td> |
|
<td style="text-align: right;">33.6</td> |
|
<td style="text-align: right;">18.3</td> |
|
<td style="text-align: right;">23.2</td> |
|
<td style="text-align: right;">21.9</td> |
|
<td style="text-align: right;">21.1</td> |
|
</tr> |
|
<tr> |
|
<td rowspan="3">LoRA Fusion</td> |
|
<td style="text-align: right;">12.8</td> |
|
<td style="text-align: right;">10.4</td> |
|
<td style="text-align: right;">18.4</td> |
|
<td style="text-align: right;">13.9</td> |
|
<td style="text-align: right;">17.1</td> |
|
<td style="text-align: right;">17.7</td> |
|
<td style="text-align: right;">18.3</td> |
|
<td style="text-align: right;">17.7</td> |
|
</tr> |
|
<tr> |
|
<td style="text-align: right;">20.8</td> |
|
<td style="text-align: right;">28.4</td> |
|
<td style="text-align: right;">36.8</td> |
|
<td style="text-align: right;">28.7</td> |
|
<td style="text-align: right;">19.5</td> |
|
<td style="text-align: right;">21.3</td> |
|
<td style="text-align: right;">20.1</td> |
|
<td style="text-align: right;">20.3</td> |
|
</tr> |
|
<tr> |
|
<td style="text-align: right;"><strong>33.2</strong></td> |
|
<td style="text-align: right;"><strong>37.6</strong></td> |
|
<td style="text-align: right;"><strong>42.0</strong></td> |
|
<td style="text-align: right;"><strong>37.6</strong></td> |
|
<td style="text-align: right;"><strong>20.7</strong></td> |
|
<td style="text-align: right;"><strong>23.8</strong></td> |
|
<td style="text-align: right;"><strong>23.2</strong></td> |
|
<td style="text-align: right;"><strong>22.6</strong></td> |
|
</tr> |
|
</tbody> |
|
</table> |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# Citation |
|
```bibtex |
|
@inproceedings{wang-etal-2024-lora-flow, |
|
title = "LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks", |
|
author = "Wang, Hanqing and |
|
Ping, Bowen and |
|
Wang, Shuo and |
|
Han, Xu and |
|
Chen, Yun and |
|
Liu, Zhiyuan and |
|
Sun, Maosong", |
|
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)", |
|
month = aug, |
|
year = "2024", |
|
address = "Bangkok, Thailand", |
|
publisher = "Association for Computational Linguistics", |
|
url = "https://aclanthology.org/2024.acl-long.695", |
|
doi = "10.18653/v1/2024.acl-long.695", |
|
pages = "12871--12882" |
|
} |
|
``` |
|
<!-- [LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks](https://aclanthology.org/2024.acl-long.695) --> |
