Upload 13 files

README.md

@@ -5,57 +5,111 @@ base_model: agentlans/deberta-v3-xsmall-zyda-2
 tags:
 - generated_from_trainer
 model-index:
-- name: deberta-v3-xsmall-zyda-2-readability
+- name: deberta-v3-xsmall-zyda-2-transformed-readability-new
   results: []
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-# deberta-v3-xsmall-zyda-2-readability
-
-This model is a fine-tuned version of [agentlans/deberta-v3-xsmall-zyda-2](https://huggingface.co/agentlans/deberta-v3-xsmall-zyda-2) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.7798
-- Mse: 0.7798
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- learning_rate: 5e-05
-- train_batch_size: 64
-- eval_batch_size: 8
-- seed: 42
-- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: linear
-- num_epochs: 3.0
-
-### Training results
-
-| Training Loss | Epoch | Step  | Validation Loss | Mse    |
+# deberta-v3-xsmall-zyda-2-transformed-readability-new
+
+## Model Overview
+
+This model is a fine-tuned version of [agentlans/deberta-v3-xsmall-zyda-2](https://huggingface.co/agentlans/deberta-v3-xsmall-zyda-2) designed to predict text readability. It achieves the following results on the evaluation set:
+- Loss: 0.0273
+- MSE: 0.0273
+
+## Dataset Description
+
+The [dataset used for training](https://huggingface.co/datasets/agentlans/readability) comprises approximately 800&thinsp;000 paragraphs with corresponding readability metrics from four diverse sources:
+
+1. HuggingFace's Fineweb-Edu
+2. Ronen Eldan's TinyStories
+3. Wikipedia-2023-11-embed-multilingual-v3 (English only)
+4. ArXiv Abstracts-2021
+
+- **Text Length**: 50 to 2000 characters per paragraph
+- **Readability Grade**: Median of six readability metrics (Flesch-Kincaid, Gunning Fog, SMOG, Automated Readability Index, Coleman-Liau, Linsear Write)
+
+### [Data Transformation](https://huggingface.co/datasets/agentlans/text-stats#readability-score-calculation)
+- U.S. reading grade levels were transformed using the Box-Cox method (λ = 0.8766912)
+- Standardization and scale inversion were applied to generate 'readability' scores
+- Higher scores indicate easier readability
+
+### Transformation Statistics
+- λ (lambda) = 0.8766912
+- Mean (before standardization) = 7.908629
+- Standard deviation (before standardization) = 3.339119
+
+## Usage Example
+
+```python
+import torch
+import numpy as np
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+# Device setup
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+# Load model and tokenizer
+model_name = "agentlans/deberta-v3-xsmall-zyda-2-readability"
+model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=1).to(device)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+# Prediction function
+def predict_score(text):
+    inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True).to(device)
+    with torch.no_grad():
+        logits = model(**inputs).logits
+    return logits.item()
+
+# Grade level conversion function
+def grade_level(y):
+    lambda_, mean, sd = 0.8766912, 7.908629, 3.339119
+    y_unstd = (-y) * sd + mean
+    return np.power((y_unstd * lambda_ + 1), (1 / lambda_))
+
+# Example
+input_text = "The mitochondria is the powerhouse of the cell."
+readability = predict_score(input_text)
+grade = grade_level(readability)
+print(f"Predicted score: {readability:.2f}\nGrade: {grade:.1f}")
+```
+
+## Sample Outputs
+
+| Text | Readability | Grade |
+|------|------------:|------:|
+| I like to eat apples. | 2.21 | 1.6 |
+| The cat is on the mat. | 2.17 | 1.7 |
+| Birds are singing in the trees. | 2.05 | 2.1 |
+| The sun is shining brightly today. | 1.95 | 2.5 |
+| She enjoys reading books in her free time. | 1.84 | 2.9 |
+| The quick brown fox jumps over the lazy dog. | 1.75 | 3.2 |
+| After a long day at work, he finally relaxed with a cup of tea. | 1.21 | 5.4 |
+| As the storm approached, the sky turned a deep shade of gray, casting an eerie shadow over the landscape. | 0.54 | 8.2 |
+| Despite the challenges they faced, the team remained resolute in their pursuit of excellence and innovation. | -0.52 | 13.0 |
+| In a world increasingly dominated by technology, the delicate balance between human connection and digital interaction has become a focal point of contemporary discourse. | -1.91 | 19.5 |
+
+## Training Procedure
+
+### Hyperparameters
+- Learning rate: 5e-05
+- Train batch size: 64
+- Eval batch size: 8
+- Seed: 42
+- Optimizer: AdamW (betas=(0.9,0.999), epsilon=1e-08)
+- LR scheduler: Linear
+- Number of epochs: 3.0
+
+### Training Results
+
+| Training Loss | Epoch | Step  | Validation Loss | MSE    |
 |:-------------:|:-----:|:-----:|:---------------:|:------:|
-| 0.7074        | 1.0   | 13589 | 1.1765          | 1.1765 |
-| 0.5843        | 2.0   | 27178 | 0.9119          | 0.9119 |
-| 0.519         | 3.0   | 40767 | 0.7798          | 0.7798 |
-
-
-### Framework versions
-
-- Transformers 4.46.3
-- Pytorch 2.5.1+cu124
-- Datasets 3.1.0
-- Tokenizers 0.20.3
+| 0.0297        | 1.0   | 13589 | 0.0302          | 0.0302 |
+| 0.0249        | 2.0   | 27178 | 0.0279          | 0.0279 |
+| 0.0218        | 3.0   | 40767 | 0.0273          | 0.0273 |
+
+## Framework Versions
+- Transformers: 4.46.3
+- PyTorch: 2.5.1+cu124
+- Datasets: 3.1.0
+- Tokenizers: 0.20.3

all_results.json

@@ -1,15 +1,15 @@
 {
     "epoch": 3.0,
-    "eval_loss": 0.7798499464988708,
-    "eval_mse": 0.7798499701682378,
-    "eval_runtime": 50.5561,
+    "eval_loss": 0.02727937512099743,
+    "eval_mse": 0.027279374637490018,
+    "eval_runtime": 52.9942,
     "eval_samples": 50000,
-    "eval_samples_per_second": 989.0,
-    "eval_steps_per_second": 123.625,
+    "eval_samples_per_second": 943.499,
+    "eval_steps_per_second": 117.937,
     "total_flos": 4.296607448400461e+16,
-    "train_loss": 0.8136542444680382,
-    "train_runtime": 5173.0086,
+    "train_loss": 0.028245126846964373,
+    "train_runtime": 5158.77,
     "train_samples": 869663,
-    "train_samples_per_second": 504.347,
-    "train_steps_per_second": 7.881
+    "train_samples_per_second": 505.739,
+    "train_steps_per_second": 7.902
 }

eval_results.json

@@ -1,9 +1,9 @@
 {
     "epoch": 3.0,
-    "eval_loss": 0.7798499464988708,
-    "eval_mse": 0.7798499701682378,
-    "eval_runtime": 50.5561,
+    "eval_loss": 0.02727937512099743,
+    "eval_mse": 0.027279374637490018,
+    "eval_runtime": 52.9942,
     "eval_samples": 50000,
-    "eval_samples_per_second": 989.0,
-    "eval_steps_per_second": 123.625
+    "eval_samples_per_second": 943.499,
+    "eval_steps_per_second": 117.937
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:495e86a2e1dcf8d1d0785394284c87a64aa9bec1b3e12e7dea159fffb0a0b851
+oid sha256:323d9dec68b5d1b6bc4b835e4eae09de1ecd00df2e461882386614ac8e17be56
 size 283345892

train_results.json

@@ -1,9 +1,9 @@
 {
     "epoch": 3.0,
     "total_flos": 4.296607448400461e+16,
-    "train_loss": 0.8136542444680382,
-    "train_runtime": 5173.0086,
+    "train_loss": 0.028245126846964373,
+    "train_runtime": 5158.77,
     "train_samples": 869663,
-    "train_samples_per_second": 504.347,
-    "train_steps_per_second": 7.881
+    "train_samples_per_second": 505.739,
+    "train_steps_per_second": 7.902
 }

trainer_state.json

@@ -1,6 +1,6 @@
 {
-  "best_metric": 0.7798499464988708,
-  "best_model_checkpoint": "deberta-v3-xsmall-zyda-2-readability/checkpoint-40767",
+  "best_metric": 0.02727937512099743,
+  "best_model_checkpoint": "deberta-v3-xsmall-zyda-2-transformed-readability-new/checkpoint-40767",
   "epoch": 3.0,
   "eval_steps": 500,
   "global_step": 40767,
@@ -10,606 +10,606 @@
   "log_history": [
     {
       "epoch": 0.03679446611229671,
-      "grad_norm": 16.423442840576172,
+      "grad_norm": 1.7569845914840698,
       "learning_rate": 4.9386758898128385e-05,
-      "loss": 13.8081,
+      "loss": 0.0913,
       "step": 500
     },
     {
       "epoch": 0.07358893222459342,
-      "grad_norm": 30.925838470458984,
+      "grad_norm": 1.4311065673828125,
       "learning_rate": 4.877351779625678e-05,
-      "loss": 1.2516,
+      "loss": 0.0531,
       "step": 1000
     },
     {
       "epoch": 0.11038339833689013,
-      "grad_norm": 7.902581214904785,
+      "grad_norm": 0.627588152885437,
       "learning_rate": 4.8160276694385164e-05,
-      "loss": 1.0974,
+      "loss": 0.0463,
       "step": 1500
     },
     {
       "epoch": 0.14717786444918685,
-      "grad_norm": 23.696027755737305,
+      "grad_norm": 1.7137391567230225,
       "learning_rate": 4.754703559251355e-05,
-      "loss": 0.9989,
+      "loss": 0.0419,
       "step": 2000
     },
     {
       "epoch": 0.18397233056148354,
-      "grad_norm": 6.253603935241699,
+      "grad_norm": 0.31713053584098816,
       "learning_rate": 4.693379449064194e-05,
-      "loss": 0.9566,
+      "loss": 0.0404,
       "step": 2500
     },
     {
       "epoch": 0.22076679667378027,
-      "grad_norm": 15.503094673156738,
+      "grad_norm": 0.9696204662322998,
       "learning_rate": 4.632055338877033e-05,
-      "loss": 0.9294,
+      "loss": 0.0389,
       "step": 3000
     },
     {
       "epoch": 0.257561262786077,
-      "grad_norm": 10.995322227478027,
+      "grad_norm": 0.7807271480560303,
       "learning_rate": 4.570731228689872e-05,
-      "loss": 0.9277,
+      "loss": 0.0383,
       "step": 3500
     },
     {
       "epoch": 0.2943557288983737,
-      "grad_norm": 35.4528694152832,
+      "grad_norm": 0.7852717638015747,
       "learning_rate": 4.509407118502711e-05,
-      "loss": 0.8674,
+      "loss": 0.0369,
       "step": 4000
     },
     {
       "epoch": 0.3311501950106704,
-      "grad_norm": 28.436100006103516,
+      "grad_norm": 0.4416508674621582,
       "learning_rate": 4.448083008315549e-05,
-      "loss": 0.8675,
+      "loss": 0.0363,
       "step": 4500
     },
     {
       "epoch": 0.3679446611229671,
-      "grad_norm": 8.568737983703613,
+      "grad_norm": 0.6548042297363281,
       "learning_rate": 4.386758898128389e-05,
-      "loss": 0.8243,
+      "loss": 0.0347,
       "step": 5000
     },
     {
       "epoch": 0.40473912723526384,
-      "grad_norm": 11.543416976928711,
+      "grad_norm": 0.4651222825050354,
       "learning_rate": 4.325434787941227e-05,
-      "loss": 0.8138,
+      "loss": 0.0349,
       "step": 5500
     },
     {
       "epoch": 0.44153359334756054,
-      "grad_norm": 8.195497512817383,
+      "grad_norm": 0.6483996510505676,
       "learning_rate": 4.264110677754066e-05,
-      "loss": 0.7986,
+      "loss": 0.0332,
       "step": 6000
     },
     {
       "epoch": 0.47832805945985724,
-      "grad_norm": 11.104621887207031,
+      "grad_norm": 0.6971497535705566,
       "learning_rate": 4.202786567566905e-05,
-      "loss": 0.7888,
+      "loss": 0.0339,
       "step": 6500
     },
     {
       "epoch": 0.515122525572154,
-      "grad_norm": 18.536165237426758,
+      "grad_norm": 0.6818335652351379,
       "learning_rate": 4.141462457379743e-05,
-      "loss": 0.7736,
+      "loss": 0.033,
       "step": 7000
     },
     {
       "epoch": 0.5519169916844506,
-      "grad_norm": 12.101081848144531,
+      "grad_norm": 0.26788613200187683,
       "learning_rate": 4.080138347192582e-05,
-      "loss": 0.7731,
+      "loss": 0.0324,
       "step": 7500
     },
     {
       "epoch": 0.5887114577967474,
-      "grad_norm": 41.568119049072266,
+      "grad_norm": 0.5651209950447083,
       "learning_rate": 4.018814237005421e-05,
-      "loss": 0.7695,
+      "loss": 0.0327,
       "step": 8000
     },
     {
       "epoch": 0.625505923909044,
-      "grad_norm": 34.703407287597656,
+      "grad_norm": 0.526560366153717,
       "learning_rate": 3.95749012681826e-05,
-      "loss": 0.7647,
+      "loss": 0.0323,
       "step": 8500
     },
     {
       "epoch": 0.6623003900213408,
-      "grad_norm": 5.127053737640381,
+      "grad_norm": 0.5062350630760193,
       "learning_rate": 3.896166016631099e-05,
-      "loss": 0.7568,
+      "loss": 0.0317,
       "step": 9000
     },
     {
       "epoch": 0.6990948561336375,
-      "grad_norm": 23.96383285522461,
+      "grad_norm": 1.0407236814498901,
       "learning_rate": 3.834841906443938e-05,
-      "loss": 0.7503,
+      "loss": 0.0313,
       "step": 9500
     },
     {
       "epoch": 0.7358893222459342,
-      "grad_norm": 21.276172637939453,
+      "grad_norm": 0.2916816473007202,
       "learning_rate": 3.773517796256776e-05,
-      "loss": 0.7243,
+      "loss": 0.0309,
       "step": 10000
     },
     {
       "epoch": 0.7726837883582309,
-      "grad_norm": 9.888713836669922,
+      "grad_norm": 0.6440730690956116,
       "learning_rate": 3.712193686069616e-05,
-      "loss": 0.7263,
+      "loss": 0.031,
       "step": 10500
     },
     {
       "epoch": 0.8094782544705277,
-      "grad_norm": 10.007771492004395,
+      "grad_norm": 0.3350902199745178,
       "learning_rate": 3.650869575882454e-05,
-      "loss": 0.7163,
+      "loss": 0.03,
       "step": 11000
     },
     {
       "epoch": 0.8462727205828243,
-      "grad_norm": 10.484025955200195,
+      "grad_norm": 0.40470924973487854,
       "learning_rate": 3.589545465695293e-05,
-      "loss": 0.7059,
+      "loss": 0.0303,
       "step": 11500
     },
     {
       "epoch": 0.8830671866951211,
-      "grad_norm": 8.901969909667969,
+      "grad_norm": 0.5961210131645203,
       "learning_rate": 3.528221355508132e-05,
-      "loss": 0.6982,
+      "loss": 0.0298,
       "step": 12000
     },
     {
       "epoch": 0.9198616528074177,
-      "grad_norm": 24.126110076904297,
+      "grad_norm": 0.36755993962287903,
       "learning_rate": 3.466897245320971e-05,
-      "loss": 0.7096,
+      "loss": 0.0303,
       "step": 12500
     },
     {
       "epoch": 0.9566561189197145,
-      "grad_norm": 6.792783260345459,
+      "grad_norm": 0.3088076114654541,
       "learning_rate": 3.405573135133809e-05,
-      "loss": 0.7046,
+      "loss": 0.03,
       "step": 13000
     },
     {
       "epoch": 0.9934505850320112,
-      "grad_norm": 16.441293716430664,
+      "grad_norm": 0.4544014632701874,
       "learning_rate": 3.344249024946648e-05,
-      "loss": 0.7074,
+      "loss": 0.0297,
       "step": 13500
     },
     {
       "epoch": 1.0,
-      "eval_loss": 1.17648184299469,
-      "eval_mse": 1.1764817839641581,
-      "eval_runtime": 51.1635,
-      "eval_samples_per_second": 977.258,
-      "eval_steps_per_second": 122.157,
+      "eval_loss": 0.03021918050944805,
+      "eval_mse": 0.03021917968962894,
+      "eval_runtime": 50.6672,
+      "eval_samples_per_second": 986.831,
+      "eval_steps_per_second": 123.354,
       "step": 13589
     },
     {
       "epoch": 1.030245051144308,
-      "grad_norm": 11.408621788024902,
+      "grad_norm": 0.4768720269203186,
       "learning_rate": 3.282924914759487e-05,
-      "loss": 0.65,
+      "loss": 0.0272,
       "step": 14000
     },
     {
       "epoch": 1.0670395172566045,
-      "grad_norm": 8.84099006652832,
+      "grad_norm": 0.4547971189022064,
       "learning_rate": 3.221600804572326e-05,
-      "loss": 0.6433,
+      "loss": 0.0268,
       "step": 14500
     },
     {
       "epoch": 1.1038339833689013,
-      "grad_norm": 8.65352725982666,
+      "grad_norm": 0.3559289276599884,
       "learning_rate": 3.160276694385165e-05,
-      "loss": 0.6161,
+      "loss": 0.0259,
       "step": 15000
     },
     {
       "epoch": 1.140628449481198,
-      "grad_norm": 7.572135925292969,
+      "grad_norm": 0.5827292203903198,
       "learning_rate": 3.098952584198003e-05,
-      "loss": 0.6326,
+      "loss": 0.0267,
       "step": 15500
     },
     {
       "epoch": 1.1774229155934948,
-      "grad_norm": 6.609086513519287,
+      "grad_norm": 0.46800583600997925,
       "learning_rate": 3.0376284740108423e-05,
-      "loss": 0.6327,
+      "loss": 0.0265,
       "step": 16000
     },
     {
       "epoch": 1.2142173817057915,
-      "grad_norm": 9.266849517822266,
+      "grad_norm": 0.39976394176483154,
       "learning_rate": 2.976304363823681e-05,
-      "loss": 0.6375,
+      "loss": 0.0267,
       "step": 16500
     },
     {
       "epoch": 1.2510118478180883,
-      "grad_norm": 13.321990013122559,
+      "grad_norm": 0.3947591185569763,
       "learning_rate": 2.91498025363652e-05,
-      "loss": 0.645,
+      "loss": 0.0272,
       "step": 17000
     },
     {
       "epoch": 1.2878063139303848,
-      "grad_norm": 5.256217956542969,
+      "grad_norm": 0.2981387674808502,
       "learning_rate": 2.8536561434493587e-05,
-      "loss": 0.623,
+      "loss": 0.0262,
       "step": 17500
     },
     {
       "epoch": 1.3246007800426816,
-      "grad_norm": 12.81368637084961,
+      "grad_norm": 0.6237615346908569,
       "learning_rate": 2.7923320332621977e-05,
-      "loss": 0.6314,
+      "loss": 0.0268,
       "step": 18000
     },
     {
       "epoch": 1.3613952461549783,
-      "grad_norm": 11.609545707702637,
+      "grad_norm": 0.6127618551254272,
       "learning_rate": 2.7310079230750363e-05,
-      "loss": 0.6308,
+      "loss": 0.0265,
       "step": 18500
     },
     {
       "epoch": 1.398189712267275,
-      "grad_norm": 6.967829704284668,
+      "grad_norm": 0.4457475244998932,
       "learning_rate": 2.6696838128878755e-05,
-      "loss": 0.6245,
+      "loss": 0.0266,
       "step": 19000
     },
     {
       "epoch": 1.4349841783795716,
-      "grad_norm": 8.6737642288208,
+      "grad_norm": 0.32558730244636536,
       "learning_rate": 2.6083597027007138e-05,
-      "loss": 0.6259,
+      "loss": 0.0265,
       "step": 19500
     },
     {
       "epoch": 1.4717786444918683,
-      "grad_norm": 6.064223766326904,
+      "grad_norm": 0.321478009223938,
       "learning_rate": 2.5470355925135524e-05,
-      "loss": 0.624,
+      "loss": 0.0262,
       "step": 20000
     },
     {
       "epoch": 1.508573110604165,
-      "grad_norm": 8.55582046508789,
+      "grad_norm": 0.42566895484924316,
       "learning_rate": 2.4857114823263916e-05,
-      "loss": 0.608,
+      "loss": 0.0256,
       "step": 20500
     },
     {
       "epoch": 1.5453675767164619,
-      "grad_norm": 12.90141487121582,
+      "grad_norm": 0.382902592420578,
       "learning_rate": 2.4243873721392306e-05,
-      "loss": 0.6186,
+      "loss": 0.0264,
       "step": 21000
     },
     {
       "epoch": 1.5821620428287586,
-      "grad_norm": 12.488966941833496,
+      "grad_norm": 0.45045822858810425,
       "learning_rate": 2.3630632619520692e-05,
-      "loss": 0.601,
+      "loss": 0.0253,
       "step": 21500
     },
     {
       "epoch": 1.6189565089410554,
-      "grad_norm": 23.348468780517578,
+      "grad_norm": 0.317402184009552,
       "learning_rate": 2.301739151764908e-05,
-      "loss": 0.6045,
+      "loss": 0.0252,
       "step": 22000
     },
     {
       "epoch": 1.6557509750533521,
-      "grad_norm": 5.677742004394531,
+      "grad_norm": 0.43501928448677063,
       "learning_rate": 2.2404150415777467e-05,
-      "loss": 0.6029,
+      "loss": 0.0253,
       "step": 22500
     },
     {
       "epoch": 1.6925454411656486,
-      "grad_norm": 8.58395004272461,
+      "grad_norm": 0.20946815609931946,
       "learning_rate": 2.1790909313905856e-05,
-      "loss": 0.6035,
+      "loss": 0.0256,
       "step": 23000
     },
     {
       "epoch": 1.7293399072779454,
-      "grad_norm": 10.593660354614258,
+      "grad_norm": 0.4793068766593933,
       "learning_rate": 2.1177668212034242e-05,
-      "loss": 0.5971,
+      "loss": 0.0253,
       "step": 23500
     },
     {
       "epoch": 1.7661343733902422,
-      "grad_norm": 16.008560180664062,
+      "grad_norm": 0.5443638563156128,
       "learning_rate": 2.056442711016263e-05,
-      "loss": 0.5921,
+      "loss": 0.0253,
       "step": 24000
     },
     {
       "epoch": 1.8029288395025387,
-      "grad_norm": 34.0599365234375,
+      "grad_norm": 0.8820445537567139,
       "learning_rate": 1.995118600829102e-05,
-      "loss": 0.586,
+      "loss": 0.0248,
       "step": 24500
     },
     {
       "epoch": 1.8397233056148354,
-      "grad_norm": 20.452659606933594,
+      "grad_norm": 0.6084474921226501,
       "learning_rate": 1.933794490641941e-05,
-      "loss": 0.5962,
+      "loss": 0.0251,
       "step": 25000
     },
     {
       "epoch": 1.8765177717271322,
-      "grad_norm": 21.741378784179688,
+      "grad_norm": 0.42228248715400696,
       "learning_rate": 1.8724703804547796e-05,
-      "loss": 0.586,
+      "loss": 0.025,
       "step": 25500
     },
     {
       "epoch": 1.913312237839429,
-      "grad_norm": 9.27108097076416,
+      "grad_norm": 0.4069421887397766,
       "learning_rate": 1.8111462702676185e-05,
-      "loss": 0.5811,
+      "loss": 0.0246,
       "step": 26000
     },
     {
       "epoch": 1.9501067039517257,
-      "grad_norm": 25.552701950073242,
+      "grad_norm": 0.5044609904289246,
       "learning_rate": 1.7498221600804575e-05,
-      "loss": 0.5897,
+      "loss": 0.0249,
       "step": 26500
     },
     {
       "epoch": 1.9869011700640224,
-      "grad_norm": 7.014336585998535,
+      "grad_norm": 0.25344032049179077,
       "learning_rate": 1.688498049893296e-05,
-      "loss": 0.5843,
+      "loss": 0.0249,
       "step": 27000
     },
     {
       "epoch": 2.0,
-      "eval_loss": 0.9118738770484924,
-      "eval_mse": 0.9118738873106327,
-      "eval_runtime": 50.7686,
-      "eval_samples_per_second": 984.861,
-      "eval_steps_per_second": 123.108,
+      "eval_loss": 0.02789381518959999,
+      "eval_mse": 0.027893818228878035,
+      "eval_runtime": 58.0401,
+      "eval_samples_per_second": 861.474,
+      "eval_steps_per_second": 107.684,
       "step": 27178
     },
     {
       "epoch": 2.023695636176319,
-      "grad_norm": 6.413208961486816,
+      "grad_norm": 0.2608506679534912,
       "learning_rate": 1.627173939706135e-05,
-      "loss": 0.5492,
+      "loss": 0.023,
       "step": 27500
     },
     {
       "epoch": 2.060490102288616,
-      "grad_norm": 6.586884498596191,
+      "grad_norm": 0.25890249013900757,
       "learning_rate": 1.565849829518974e-05,
-      "loss": 0.5432,
+      "loss": 0.0226,
       "step": 28000
     },
     {
       "epoch": 2.0972845684009127,
-      "grad_norm": 17.055835723876953,
+      "grad_norm": 0.577273428440094,
       "learning_rate": 1.5045257193318127e-05,
-      "loss": 0.5393,
+      "loss": 0.0225,
       "step": 28500
     },
     {
       "epoch": 2.134079034513209,
-      "grad_norm": 13.087154388427734,
+      "grad_norm": 0.3473275303840637,
       "learning_rate": 1.4432016091446513e-05,
-      "loss": 0.5419,
+      "loss": 0.0228,
       "step": 29000
     },
     {
       "epoch": 2.1708735006255058,
-      "grad_norm": 4.83110237121582,
+      "grad_norm": 0.2392967790365219,
       "learning_rate": 1.38187749895749e-05,
-      "loss": 0.5357,
+      "loss": 0.0223,
       "step": 29500
     },
     {
       "epoch": 2.2076679667378025,
-      "grad_norm": 11.013864517211914,
+      "grad_norm": 0.38996148109436035,
       "learning_rate": 1.320553388770329e-05,
-      "loss": 0.5411,
+      "loss": 0.0225,
       "step": 30000
     },
     {
       "epoch": 2.2444624328500993,
-      "grad_norm": 21.888784408569336,
+      "grad_norm": 0.8742114305496216,
       "learning_rate": 1.2592292785831677e-05,
-      "loss": 0.5446,
+      "loss": 0.0227,
       "step": 30500
     },
     {
       "epoch": 2.281256898962396,
-      "grad_norm": 4.124610424041748,
+      "grad_norm": 0.2552475929260254,
       "learning_rate": 1.1979051683960066e-05,
-      "loss": 0.5364,
+      "loss": 0.0223,
       "step": 31000
     },
     {
       "epoch": 2.318051365074693,
-      "grad_norm": 6.587899208068848,
+      "grad_norm": 0.3917369842529297,
       "learning_rate": 1.1365810582088454e-05,
-      "loss": 0.5298,
+      "loss": 0.0222,
       "step": 31500
     },
     {
       "epoch": 2.3548458311869895,
-      "grad_norm": 4.52357816696167,
+      "grad_norm": 0.31826546788215637,
       "learning_rate": 1.0752569480216842e-05,
-      "loss": 0.5324,
+      "loss": 0.0221,
       "step": 32000
     },
     {
       "epoch": 2.3916402972992863,
-      "grad_norm": 6.613998889923096,
+      "grad_norm": 0.3128163516521454,
       "learning_rate": 1.0139328378345231e-05,
-      "loss": 0.5389,
+      "loss": 0.0226,
       "step": 32500
     },
     {
       "epoch": 2.428434763411583,
-      "grad_norm": 8.802297592163086,
+      "grad_norm": 0.4479590058326721,
       "learning_rate": 9.526087276473619e-06,
-      "loss": 0.527,
+      "loss": 0.0222,
       "step": 33000
     },
     {
       "epoch": 2.46522922952388,
-      "grad_norm": 8.77059268951416,
+      "grad_norm": 0.2909683883190155,
       "learning_rate": 8.912846174602008e-06,
-      "loss": 0.5203,
+      "loss": 0.0217,
       "step": 33500
     },
     {
       "epoch": 2.5020236956361765,
-      "grad_norm": 6.136614799499512,
+      "grad_norm": 0.30918726325035095,
       "learning_rate": 8.299605072730394e-06,
-      "loss": 0.5269,
+      "loss": 0.0222,
       "step": 34000
     },
     {
       "epoch": 2.5388181617484733,
-      "grad_norm": 6.427796363830566,
+      "grad_norm": 0.36887994408607483,
       "learning_rate": 7.686363970858783e-06,
-      "loss": 0.5215,
+      "loss": 0.0217,
       "step": 34500
     },
     {
       "epoch": 2.5756126278607696,
-      "grad_norm": 8.561346054077148,
+      "grad_norm": 0.38799503445625305,
       "learning_rate": 7.073122868987171e-06,
-      "loss": 0.5373,
+      "loss": 0.0227,
       "step": 35000
     },
     {
       "epoch": 2.6124070939730664,
-      "grad_norm": 4.508603096008301,
+      "grad_norm": 0.21477651596069336,
       "learning_rate": 6.459881767115559e-06,
-      "loss": 0.5319,
+      "loss": 0.0219,
       "step": 35500
     },
     {
       "epoch": 2.649201560085363,
-      "grad_norm": 13.598175048828125,
+      "grad_norm": 0.367152601480484,
       "learning_rate": 5.846640665243948e-06,
-      "loss": 0.524,
+      "loss": 0.0219,
       "step": 36000
     },
     {
       "epoch": 2.68599602619766,
-      "grad_norm": 4.264435291290283,
+      "grad_norm": 0.2770203649997711,
       "learning_rate": 5.233399563372335e-06,
-      "loss": 0.5221,
+      "loss": 0.022,
       "step": 36500
     },
     {
       "epoch": 2.7227904923099566,
-      "grad_norm": 7.450808525085449,
+      "grad_norm": 0.31007689237594604,
       "learning_rate": 4.620158461500724e-06,
-      "loss": 0.5295,
+      "loss": 0.022,
       "step": 37000
     },
     {
       "epoch": 2.7595849584222534,
-      "grad_norm": 9.741497039794922,
+      "grad_norm": 0.27211979031562805,
       "learning_rate": 4.006917359629112e-06,
-      "loss": 0.5195,
+      "loss": 0.0217,
       "step": 37500
     },
     {
       "epoch": 2.79637942453455,
-      "grad_norm": 14.861910820007324,
+      "grad_norm": 0.36633121967315674,
       "learning_rate": 3.3936762577575e-06,
-      "loss": 0.5245,
+      "loss": 0.022,
       "step": 38000
     },
     {
       "epoch": 2.8331738906468464,
-      "grad_norm": 9.222199440002441,
+      "grad_norm": 0.3557955324649811,
       "learning_rate": 2.7804351558858883e-06,
-      "loss": 0.5041,
+      "loss": 0.0211,
       "step": 38500
     },
     {
       "epoch": 2.869968356759143,
-      "grad_norm": 3.4334182739257812,
+      "grad_norm": 0.2332288920879364,
       "learning_rate": 2.1671940540142763e-06,
-      "loss": 0.5206,
+      "loss": 0.0218,
       "step": 39000
     },
     {
       "epoch": 2.90676282287144,
-      "grad_norm": 3.955873489379883,
+      "grad_norm": 0.179446280002594,
       "learning_rate": 1.5539529521426646e-06,
-      "loss": 0.5085,
+      "loss": 0.0213,
       "step": 39500
     },
     {
       "epoch": 2.9435572889837367,
-      "grad_norm": 23.795787811279297,
+      "grad_norm": 0.7559341192245483,
       "learning_rate": 9.407118502710525e-07,
-      "loss": 0.5159,
+      "loss": 0.0215,
       "step": 40000
     },
     {
       "epoch": 2.9803517550960335,
-      "grad_norm": 5.348132610321045,
+      "grad_norm": 0.2412876933813095,
       "learning_rate": 3.2747074839944075e-07,
-      "loss": 0.519,
+      "loss": 0.0218,
       "step": 40500
     },
     {
       "epoch": 3.0,
-      "eval_loss": 0.7798499464988708,
-      "eval_mse": 0.7798499701682378,
-      "eval_runtime": 50.9671,
-      "eval_samples_per_second": 981.024,
-      "eval_steps_per_second": 122.628,
+      "eval_loss": 0.02727937512099743,
+      "eval_mse": 0.027279374637490018,
+      "eval_runtime": 54.5217,
+      "eval_samples_per_second": 917.066,
+      "eval_steps_per_second": 114.633,
       "step": 40767
     },
     {
       "epoch": 3.0,
       "step": 40767,
       "total_flos": 4.296607448400461e+16,
-      "train_loss": 0.8136542444680382,
-      "train_runtime": 5173.0086,
-      "train_samples_per_second": 504.347,
-      "train_steps_per_second": 7.881
+      "train_loss": 0.028245126846964373,
+      "train_runtime": 5158.77,
+      "train_samples_per_second": 505.739,
+      "train_steps_per_second": 7.902
     }
   ],
   "logging_steps": 500,

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5cbf596800b58b777319a4106900f8b91d718d2aae2495835c9470de34720bc4
+oid sha256:f30273452aa77f30718399ec672e7077759fe98f5d2ccfd61089212882abb1bc
 size 5368