Update README.md
Browse files
README.md
CHANGED
@@ -155,10 +155,10 @@ and tokenized with a maximum length of 512 for inputs and 256 for responses usin
|
|
155 |
- **LR Scheduler:** Cosine scheduler with a warmup ratio of 10%
|
156 |
- **Quantization:** 4-bit NF4 (with double quantization) using `torch.bfloat16`
|
157 |
- **LoRA Parameters:**
|
158 |
-
|
159 |
-
|
160 |
-
|
161 |
-
|
162 |
- **Checkpointing:**
|
163 |
Model saved at the end of every epoch
|
164 |
Early stopping with a patience of 2 epochs based on evaluation loss
|
|
|
155 |
- **LR Scheduler:** Cosine scheduler with a warmup ratio of 10%
|
156 |
- **Quantization:** 4-bit NF4 (with double quantization) using `torch.bfloat16`
|
157 |
- **LoRA Parameters:**
|
158 |
+
**Rank (r):** 32
|
159 |
+
**Alpha:** 64
|
160 |
+
**Dropout:** 0.1
|
161 |
+
**Target Modules:** `["q", "v"]`
|
162 |
- **Checkpointing:**
|
163 |
Model saved at the end of every epoch
|
164 |
Early stopping with a patience of 2 epochs based on evaluation loss
|