fine-tuning-Phi2-with-webglm-qa-with-lora

README.md

@@ -16,7 +16,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [microsoft/phi-2](https://huggingface.co/microsoft/phi-2) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 1.0306
+- Loss: 0.5112
 
 ## Model description
 
@@ -43,19 +43,19 @@ The following hyperparameters were used during training:
 - total_train_batch_size: 10
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: linear
-- lr_scheduler_warmup_steps: 20
-- training_steps: 100
+- lr_scheduler_warmup_steps: 10
+- training_steps: 50
 - mixed_precision_training: Native AMP
 
 ### Training results
 
 | Training Loss | Epoch | Step | Validation Loss |
 |:-------------:|:-----:|:----:|:---------------:|
-| 4.3301        | 0.2   | 20   | 2.6927          |
-| 1.4823        | 0.4   | 40   | 1.1116          |
-| 1.1068        | 0.6   | 60   | 1.0615          |
-| 1.0998        | 0.8   | 80   | 1.0379          |
-| 1.0673        | 1.0   | 100  | 1.0306          |
+| 8.0656        | 0.2   | 10   | 6.1207          |
+| 3.4473        | 0.4   | 20   | 0.9409          |
+| 0.6293        | 0.6   | 30   | 0.5648          |
+| 0.455         | 0.8   | 40   | 0.5257          |
+| 0.4325        | 1.0   | 50   | 0.5112          |
 
 
 ### Framework versions

adapter_config.json

@@ -19,12 +19,12 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "q_proj",
     "k_proj",
-    "dense",
     "v_proj",
     "fc1",
-    "fc2"
+    "fc2",
+    "q_proj",
+    "dense"
   ],
   "task_type": "CAUSAL_LM"
 }

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ed1c26a756f58dce46d78268975f6d9cc1a0d6c13d84403cbfe3e33393551d3e
+oid sha256:20b26d06164cbc59c6eb5c608e01279ec6aafc6e98021551dc840350210cb582
 size 94422368

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4e4274db79a7ba2791d4ccd7e4b9b44bfbf36016c11dad1e53f6dbb0f5900e2b
+oid sha256:9e5612a777b0fa79303c9a2432f3a86539e7e19bb8723dde492cfaf93aff3767
 size 4283