Fine-tuned smolLM2-135M with LoRA on camel-ai/physics

README.md

@@ -14,9 +14,9 @@ should probably proofread and complete it, then remove this comment. -->
 
 # fine-tuned-smolLM2-135M-with-LoRA-on-camel-ai-physics
 
-This model is a fine-tuned version of [HuggingFaceTB/SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) on an camel-ai/physics dataset.
+This model is a fine-tuned version of [HuggingFaceTB/SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 1.0047
+- Loss: 0.9705
 
 ## Model description
 
@@ -40,16 +40,21 @@ The following hyperparameters were used during training:
 - eval_batch_size: 4
 - seed: 42
 - optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: linear
-- num_epochs: 3
+- lr_scheduler_type: cosine
+- num_epochs: 8
 
 ### Training results
 
 | Training Loss | Epoch | Step  | Validation Loss |
 |:-------------:|:-----:|:-----:|:---------------:|
-| 1.0162        | 1.0   | 4000  | 1.0420          |
-| 1.0262        | 2.0   | 8000  | 1.0134          |
-| 1.0067        | 3.0   | 12000 | 1.0047          |
+| 1.0142        | 1.0   | 4000  | 1.0403          |
+| 1.0231        | 2.0   | 8000  | 1.0082          |
+| 0.9995        | 3.0   | 12000 | 0.9918          |
+| 0.9527        | 4.0   | 16000 | 0.9822          |
+| 0.9351        | 5.0   | 20000 | 0.9752          |
+| 0.9126        | 6.0   | 24000 | 0.9719          |
+| 0.9161        | 7.0   | 28000 | 0.9706          |
+| 0.9194        | 8.0   | 32000 | 0.9705          |
 
 
 ### Framework versions

adapter_config.json

@@ -20,8 +20,8 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "v_proj",
-    "q_proj"
+    "q_proj",
+    "v_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:05385138e4a73fe2f6d0e135631ba499d2670c47ad54d702182a1b7aa07a74e1
+oid sha256:059bd52dfe5c707398401b4deb83345fc1309ff04e1fbd08c5cef6559d948806
 size 3702168

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d4a15d761e2a319ab5a1242d68e02509d0416bef1b2e9f394a75f923744fe76a
+oid sha256:51229cc4a399f51e7c3700b7bdc293400663fa6083fd49be32009de37d362d8b
 size 5240