license: cc-by-nc-4.0
pretty_name: C
extra_gated_fields:
Full Name: text
Affiliation (Organization/University): text
Designation/Status in Your Organization: text
Country: country
I want to use this model for (please provide the reason(s)): text
LoRMA model is free for research use but NOT for commercial use; do you agree if you are provided with the LoRMA model, you will NOT use for any commercial purposes: checkbox
LoRMA: Low-Rank Multiplicative Adaptation for LLMs
Title
LoRMA: Low-Rank Multiplicative Adaptation for LLMs
Abstract
Large Language Models have shown remarkable capabilities in the NLP domain.Their effectiveness can mainly be attributed to their ability to adapt to an array of downstream tasks. However, generally, full fine-tuning is a computationally expensive job. To mitigate this, many techniques have been developed that prime efficiency, a prominent one being Low-Rank Adaptation (LoRA). However, LoRA and its variants employ re-parametrized additive updates. In this paper, we propose Low-Rank Multiplicative Adaptation (LoRMA), which shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations. We tackle challenges such as computational complexity and rank bottleneck of matrix multiplication by effectively re-ordering operations and introducing rank inflation strategies. We conduct extensive experiments to demonstrate the effectiveness of our approach in terms of various evaluation metrics.
For more details: