Fix Lora config error for Llama3 (#1659)
Browse filesThe current yml code throws an error: ValueError: Please set lora_modules_to_save to [`embed_tokens`, `lm_head`] when using an adapter and changing the special tokens.
I added the required changes to resolve it
examples/llama-3/lora-8b.yml
CHANGED
|
@@ -24,6 +24,9 @@ lora_alpha: 16
|
|
| 24 |
lora_dropout: 0.05
|
| 25 |
lora_target_linear: true
|
| 26 |
lora_fan_in_fan_out:
|
|
|
|
|
|
|
|
|
|
| 27 |
|
| 28 |
wandb_project:
|
| 29 |
wandb_entity:
|
|
|
|
| 24 |
lora_dropout: 0.05
|
| 25 |
lora_target_linear: true
|
| 26 |
lora_fan_in_fan_out:
|
| 27 |
+
lora_modules_to_save:
|
| 28 |
+
- embed_tokens
|
| 29 |
+
- lm_head
|
| 30 |
|
| 31 |
wandb_project:
|
| 32 |
wandb_entity:
|