PLM-Secondary-Structure-Generation

README.md

@@ -15,8 +15,8 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [ElnaggarLab/ankh-base](https://huggingface.co/ElnaggarLab/ankh-base) on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.1736
-- Q3 Accuracy: 0.5107
+- Loss: 0.1649
+- Q3 Accuracy: 0.5470
 
 ## Model description
 
@@ -35,27 +35,36 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 1e-05
+- learning_rate: 0.001
 - train_batch_size: 1
 - eval_batch_size: 8
-- seed: 7
+- seed: 42
 - distributed_type: multi-GPU
-- num_devices: 6
-- gradient_accumulation_steps: 2
-- total_train_batch_size: 12
-- total_eval_batch_size: 48
+- num_devices: 4
+- gradient_accumulation_steps: 4
+- total_train_batch_size: 16
+- total_eval_batch_size: 32
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: linear
 - lr_scheduler_warmup_ratio: 0.3
-- num_epochs: 3
+- num_epochs: 1
 
 ### Training results
 
 | Training Loss | Epoch | Step | Validation Loss | Q3 Accuracy |
 |:-------------:|:-----:|:----:|:---------------:|:-----------:|
-| 0.2839        | 1.0   | 848  | 0.2462          | 0.3434      |
-| 0.18          | 2.0   | 1696 | 0.1814          | 0.5245      |
-| 0.1691        | 3.0   | 2544 | 0.1736          | 0.5107      |
+| 1.0993        | 0.08  | 50   | 0.3957          | 0.4526      |
+| 0.4077        | 0.16  | 100  | 0.3564          | 0.5574      |
+| 0.3475        | 0.24  | 150  | 0.3082          | 0.4327      |
+| 0.2474        | 0.31  | 200  | 0.2171          | 0.5477      |
+| 0.2567        | 0.39  | 250  | 0.3058          | 0.4924      |
+| 0.2948        | 0.47  | 300  | 0.1828          | 0.5470      |
+| 0.2148        | 0.55  | 350  | 0.1873          | 0.5470      |
+| 0.209         | 0.63  | 400  | 0.1832          | 0.5470      |
+| 0.1761        | 0.71  | 450  | 0.1791          | 0.5103      |
+| 0.1791        | 0.79  | 500  | 0.1700          | 0.5470      |
+| 0.1641        | 0.87  | 550  | 0.1684          | 0.5470      |
+| 0.1653        | 0.94  | 600  | 0.1649          | 0.5470      |
 
 
 ### Framework versions

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f03af4e88b7f1d44444e0af8f6791129801b996e858a68b64b7b27e44b9c7d26
+oid sha256:18fdf824df75a1a0b4b3b77df233d6eb7d72ba594951b3071eda50b2b6bbd0b0
 size 2946957610

tokenizer.json

@@ -1,7 +1,21 @@
 {
   "version": "1.0",
-  "truncation": null,
-  "padding": null,
+  "truncation": {
+    "direction": "Right",
+    "max_length": 825,
+    "strategy": "LongestFirst",
+    "stride": 0
+  },
+  "padding": {
+    "strategy": {
+      "Fixed": 825
+    },
+    "direction": "Right",
+    "pad_to_multiple_of": null,
+    "pad_id": 0,
+    "pad_type_id": 0,
+    "pad_token": "<pad>"
+  },
   "added_tokens": [
     {
       "id": 0,

tokenizer_config.json

@@ -1078,5 +1078,6 @@
   "model_max_length": 1000000000000000019884624838656,
   "pad_token": "<pad>",
   "tokenizer_class": "T5Tokenizer",
-  "unk_token": "<unk>"
+  "unk_token": "<unk>",
+  "use_fast": true
 }

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:92e4538ec9d43680ca4f10ea9901404682f8865f418aa8b7cb9878a2bf913d7c
+oid sha256:52c0ad81e3b3de1651910d8e6ca794e1eef1803026e9e4eb120b48685a1838ff
 size 5880