Update README.md
Browse files
README.md
CHANGED
|
@@ -1,2 +1,12 @@
|
|
| 1 |
-
|
| 2 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
---
|
| 4 |
+
We released all of our checkpoints used in [LoRA-Flow](https://aclanthology.org/2024.acl-long.695.pdf) which has been accepted to ACL 2024 main conference.
|
| 5 |
+
# Summary
|
| 6 |
+
In this repo, we release LoRA and the gate of 7B models trained in our paper in HuggingFace format.
|
| 7 |
+
# Method
|
| 8 |
+
The following picture has shown our proposed method, we use layer-wise fusion gates to facilitate dynamic LoRA fusion, which project input hidden states of each layer into fusion weights.
|
| 9 |
+
|
| 10 |
+
# Citation
|
| 11 |
+
if you find our repo is helpful, please cite the following
|
| 12 |
+
[LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks](https://aclanthology.org/2024.acl-long.695)
|