Upload 13 files

README.md

@@ -4,58 +4,99 @@ license: mit
 base_model: agentlans/deberta-v3-xsmall-zyda-2
 tags:
 - generated_from_trainer
+- sentiment-analysis
+- twitter-sentiment
 model-index:
-- name: deberta-v3-xsmall-zyda-2-sentiment
+- name: deberta-v3-xsmall-zyda-2-transformed-sentiment-new
   results: []
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# DeBERTa-v3-XSmall Sentiment Analysis Model
 
-# deberta-v3-xsmall-zyda-2-sentiment
+## Model Overview
 
-This model is a fine-tuned version of [agentlans/deberta-v3-xsmall-zyda-2](https://huggingface.co/agentlans/deberta-v3-xsmall-zyda-2) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.0493
-- Mse: 0.0493
+This model is a fine-tuned version of [agentlans/deberta-v3-xsmall-zyda-2](https://huggingface.co/agentlans/deberta-v3-xsmall-zyda-2) optimized for sentiment analysis on Twitter data. It achieves the following results on the evaluation set:
 
-## Model description
+- Loss: 0.0656
+- MSE: 0.0656
 
-More information needed
+## Dataset
 
-## Intended uses & limitations
+The model was trained on the [Twitter Sentiment Meta-Analysis Dataset](https://huggingface.co/datasets/agentlans/twitter-sentiment-meta-analysis).
 
-More information needed
+### Dataset Description
 
-## Training and evaluation data
+This dataset contains sentiment analysis results for English tweets collected between September 2009 and January 2010. The tweets were processed and analyzed using 10 different sentiment classifiers, with the final sentiment score derived from principal component analysis (PCA).
 
-More information needed
+- **Source**: Cheng-Caverlee-Lee Twitter Scrape (Sept 2009 - Jan 2010)
+- **Size**: 138,690 tweets
+- **Language**: English only (filtered using langdetect)
 
-## Training procedure
+## Usage
 
-### Training hyperparameters
+Here's an example of how to use the model for sentiment prediction:
 
-The following hyperparameters were used during training:
-- learning_rate: 5e-05
-- train_batch_size: 64
-- eval_batch_size: 8
-- seed: 42
-- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: linear
-- num_epochs: 3.0
+```
+import torch
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
 
-### Training results
+# Load model and tokenizer
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+model_name = "agentlans/deberta-v3-xsmall-zyda-2-sentiment"
+model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=1).to(device)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
 
-| Training Loss | Epoch | Step | Validation Loss | Mse    |
-|:-------------:|:-----:|:----:|:---------------:|:------:|
-| 0.0627        | 1.0   | 3143 | 0.0665          | 0.0665 |
-| 0.0411        | 2.0   | 6286 | 0.0493          | 0.0493 |
-| 0.0321        | 3.0   | 9429 | 0.0524          | 0.0524 |
+# Function to perform inference
+def predict_score(text):
+    inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True).to(device)
+    with torch.no_grad():
+        logits = model(**inputs).logits
+    return logits.item()
+
+# Example usage
+input_text = "I accidentally the whole thing. Is that bad?"
+score = predict_score(input_text)
+print(f"Predicted score: {score}")
+```
+
+## Example Outputs
+
+| Text | Sentiment |
+|------|----------:|
+| Nothing seems to go right, and I'm constantly frustrated. | -2.25 |
+| Everything is falling apart, and I can't see any way out. | -2.02 |
+| I feel completely overwhelmed by the challenges I face. | -1.62 |
+| There are some minor improvements, but overall, things are still tough. | -0.81 |
+| I can see a glimmer of hope amidst the difficulties I encounter. | 1.03 |
+| Things are starting to look up, and I'm cautiously optimistic. | 2.06 |
+| There are many good things happening, and I appreciate them. | 2.23 |
+| I'm feeling more positive about my situation than I have in a while. | 2.39 |
+| Every day brings new joy and possibilities; I feel truly blessed. | 2.54 |
+| Life is full of opportunities, and I'm excited about the future. | 2.56 |
+
+## Training Procedure
 
+### Hyperparameters
+
+- Learning rate: 5e-05
+- Train batch size: 64
+- Eval batch size: 8
+- Seed: 42
+- Optimizer: AdamW with betas=(0.9, 0.999) and epsilon=1e-08
+- LR scheduler: Linear
+- Number of epochs: 3.0
+
+### Training Results
+
+| Training Loss | Epoch | Step | Validation Loss | MSE    |
+|:-------------:|:-----:|:----:|:---------------:|:------:|
+| 0.0792        | 1.0   | 2011 | 0.0871          | 0.0871 |
+| 0.0541        | 2.0   | 4022 | 0.0691          | 0.0691 |
+| 0.0411        | 3.0   | 6033 | 0.0656          | 0.0656 |
 
-### Framework versions
+## Framework Versions
 
-- Transformers 4.46.3
-- Pytorch 2.5.1+cu124
-- Datasets 3.1.0
-- Tokenizers 0.20.3
+- Transformers: 4.46.3
+- PyTorch: 2.5.1+cu124
+- Datasets: 3.1.0
+- Tokenizers: 0.20.3

all_results.json

@@ -1,15 +1,15 @@
 {
     "epoch": 3.0,
-    "eval_loss": 0.04927213117480278,
-    "eval_mse": 0.049272132016595305,
-    "eval_runtime": 10.4326,
+    "eval_loss": 0.06556913256645203,
+    "eval_mse": 0.06556913494220615,
+    "eval_runtime": 13.1744,
     "eval_samples": 10000,
-    "eval_samples_per_second": 958.536,
-    "eval_steps_per_second": 119.817,
-    "total_flos": 9935679003367680.0,
-    "train_loss": 0.05866297316179509,
-    "train_runtime": 1207.603,
-    "train_samples": 201105,
-    "train_samples_per_second": 499.597,
-    "train_steps_per_second": 7.808
+    "eval_samples_per_second": 759.049,
+    "eval_steps_per_second": 94.881,
+    "total_flos": 6357984788759040.0,
+    "train_loss": 0.07220485827706652,
+    "train_runtime": 846.782,
+    "train_samples": 128690,
+    "train_samples_per_second": 455.926,
+    "train_steps_per_second": 7.125
 }

eval_results.json

@@ -1,9 +1,9 @@
 {
     "epoch": 3.0,
-    "eval_loss": 0.04927213117480278,
-    "eval_mse": 0.049272132016595305,
-    "eval_runtime": 10.4326,
+    "eval_loss": 0.06556913256645203,
+    "eval_mse": 0.06556913494220615,
+    "eval_runtime": 13.1744,
     "eval_samples": 10000,
-    "eval_samples_per_second": 958.536,
-    "eval_steps_per_second": 119.817
+    "eval_samples_per_second": 759.049,
+    "eval_steps_per_second": 94.881
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:64bb0296f7b370ce35b0666cd7f26fb0bb06c64245a16e871885925e52a90f49
+oid sha256:2ca36fde7f77cd9138373636d634d704dc626ed3f64e5adca78c6790760099f0
 size 283345892

train_results.json

@@ -1,9 +1,9 @@
 {
     "epoch": 3.0,
-    "total_flos": 9935679003367680.0,
-    "train_loss": 0.05866297316179509,
-    "train_runtime": 1207.603,
-    "train_samples": 201105,
-    "train_samples_per_second": 499.597,
-    "train_steps_per_second": 7.808
+    "total_flos": 6357984788759040.0,
+    "train_loss": 0.07220485827706652,
+    "train_runtime": 846.782,
+    "train_samples": 128690,
+    "train_samples_per_second": 455.926,
+    "train_steps_per_second": 7.125
 }

trainer_state.json

@@ -1,178 +1,136 @@
 {
-  "best_metric": 0.04927213117480278,
-  "best_model_checkpoint": "deberta-v3-xsmall-zyda-2-sentiment/checkpoint-6286",
+  "best_metric": 0.06556913256645203,
+  "best_model_checkpoint": "deberta-v3-xsmall-zyda-2-transformed-sentiment-new/checkpoint-6033",
   "epoch": 3.0,
   "eval_steps": 500,
-  "global_step": 9429,
+  "global_step": 6033,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
   "log_history": [
     {
-      "epoch": 0.1590836780146357,
-      "grad_norm": 1.8468247652053833,
-      "learning_rate": 4.7348605366422736e-05,
-      "loss": 0.2115,
+      "epoch": 0.2486325211337643,
+      "grad_norm": 2.0000367164611816,
+      "learning_rate": 4.5856124647770596e-05,
+      "loss": 0.2003,
       "step": 500
     },
     {
-      "epoch": 0.3181673560292714,
-      "grad_norm": 1.7370903491973877,
-      "learning_rate": 4.4697210732845476e-05,
-      "loss": 0.101,
+      "epoch": 0.4972650422675286,
+      "grad_norm": 2.3387935161590576,
+      "learning_rate": 4.17122492955412e-05,
+      "loss": 0.1052,
       "step": 1000
     },
     {
-      "epoch": 0.4772510340439071,
-      "grad_norm": 1.7206146717071533,
-      "learning_rate": 4.2045816099268216e-05,
-      "loss": 0.0846,
+      "epoch": 0.7458975634012929,
+      "grad_norm": 1.853918194770813,
+      "learning_rate": 3.7568373943311785e-05,
+      "loss": 0.085,
       "step": 1500
     },
     {
-      "epoch": 0.6363347120585428,
-      "grad_norm": 1.1373802423477173,
-      "learning_rate": 3.9394421465690956e-05,
-      "loss": 0.0748,
+      "epoch": 0.9945300845350572,
+      "grad_norm": 1.7671293020248413,
+      "learning_rate": 3.342449859108238e-05,
+      "loss": 0.0792,
       "step": 2000
     },
     {
-      "epoch": 0.7954183900731785,
-      "grad_norm": 0.9603880047798157,
-      "learning_rate": 3.674302683211369e-05,
-      "loss": 0.0691,
+      "epoch": 1.0,
+      "eval_loss": 0.08709739148616791,
+      "eval_mse": 0.08709739712527088,
+      "eval_runtime": 14.8419,
+      "eval_samples_per_second": 673.767,
+      "eval_steps_per_second": 84.221,
+      "step": 2011
+    },
+    {
+      "epoch": 1.2431626056688214,
+      "grad_norm": 1.0026581287384033,
+      "learning_rate": 2.928062323885298e-05,
+      "loss": 0.0594,
       "step": 2500
     },
     {
-      "epoch": 0.9545020680878142,
-      "grad_norm": 1.0165342092514038,
-      "learning_rate": 3.409163219853643e-05,
-      "loss": 0.0627,
+      "epoch": 1.4917951268025857,
+      "grad_norm": 0.9303980469703674,
+      "learning_rate": 2.5136747886623573e-05,
+      "loss": 0.0594,
       "step": 3000
     },
     {
-      "epoch": 1.0,
-      "eval_loss": 0.06652908027172089,
-      "eval_mse": 0.06652908171153529,
-      "eval_runtime": 10.5244,
-      "eval_samples_per_second": 950.17,
-      "eval_steps_per_second": 118.771,
-      "step": 3143
-    },
-    {
-      "epoch": 1.1135857461024499,
-      "grad_norm": 0.9926055073738098,
-      "learning_rate": 3.144023756495917e-05,
-      "loss": 0.0522,
+      "epoch": 1.74042764793635,
+      "grad_norm": 1.7368980646133423,
+      "learning_rate": 2.0992872534394168e-05,
+      "loss": 0.0551,
       "step": 3500
     },
     {
-      "epoch": 1.2726694241170855,
-      "grad_norm": 1.247205376625061,
-      "learning_rate": 2.878884293138191e-05,
-      "loss": 0.0485,
+      "epoch": 1.9890601690701144,
+      "grad_norm": 0.6475295424461365,
+      "learning_rate": 1.684899718216476e-05,
+      "loss": 0.0541,
       "step": 4000
     },
     {
-      "epoch": 1.4317531021317214,
-      "grad_norm": 1.7589031457901,
-      "learning_rate": 2.6137448297804644e-05,
-      "loss": 0.0463,
+      "epoch": 2.0,
+      "eval_loss": 0.06912554055452347,
+      "eval_mse": 0.06912553393413896,
+      "eval_runtime": 13.2293,
+      "eval_samples_per_second": 755.898,
+      "eval_steps_per_second": 94.487,
+      "step": 4022
+    },
+    {
+      "epoch": 2.2376926902038785,
+      "grad_norm": 0.6805059909820557,
+      "learning_rate": 1.2705121829935357e-05,
+      "loss": 0.0444,
       "step": 4500
     },
     {
-      "epoch": 1.590836780146357,
-      "grad_norm": 0.7484694719314575,
-      "learning_rate": 2.3486053664227384e-05,
-      "loss": 0.0443,
+      "epoch": 2.486325211337643,
+      "grad_norm": 1.3735737800598145,
+      "learning_rate": 8.56124647770595e-06,
+      "loss": 0.043,
       "step": 5000
     },
     {
-      "epoch": 1.7499204581609926,
-      "grad_norm": 1.5068027973175049,
-      "learning_rate": 2.083465903065012e-05,
-      "loss": 0.0421,
+      "epoch": 2.734957732471407,
+      "grad_norm": 0.9396611452102661,
+      "learning_rate": 4.417371125476545e-06,
+      "loss": 0.0422,
       "step": 5500
     },
     {
-      "epoch": 1.9090041361756285,
-      "grad_norm": 0.832625150680542,
-      "learning_rate": 1.818326439707286e-05,
+      "epoch": 2.9835902536051715,
+      "grad_norm": 0.756208062171936,
+      "learning_rate": 2.7349577324714074e-07,
       "loss": 0.0411,
       "step": 6000
     },
-    {
-      "epoch": 2.0,
-      "eval_loss": 0.04927213117480278,
-      "eval_mse": 0.049272132016595305,
-      "eval_runtime": 11.3101,
-      "eval_samples_per_second": 884.162,
-      "eval_steps_per_second": 110.52,
-      "step": 6286
-    },
-    {
-      "epoch": 2.068087814190264,
-      "grad_norm": 0.6708300709724426,
-      "learning_rate": 1.5531869763495598e-05,
-      "loss": 0.0387,
-      "step": 6500
-    },
-    {
-      "epoch": 2.2271714922048997,
-      "grad_norm": 0.6490187644958496,
-      "learning_rate": 1.2880475129918337e-05,
-      "loss": 0.0337,
-      "step": 7000
-    },
-    {
-      "epoch": 2.3862551702195356,
-      "grad_norm": 0.7127770185470581,
-      "learning_rate": 1.0229080496341075e-05,
-      "loss": 0.0324,
-      "step": 7500
-    },
-    {
-      "epoch": 2.545338848234171,
-      "grad_norm": 0.6604452133178711,
-      "learning_rate": 7.5776858627638146e-06,
-      "loss": 0.0326,
-      "step": 8000
-    },
-    {
-      "epoch": 2.704422526248807,
-      "grad_norm": 0.5042712092399597,
-      "learning_rate": 4.926291229186552e-06,
-      "loss": 0.0323,
-      "step": 8500
-    },
-    {
-      "epoch": 2.8635062042634427,
-      "grad_norm": 0.573316752910614,
-      "learning_rate": 2.2748965956092908e-06,
-      "loss": 0.0321,
-      "step": 9000
-    },
     {
       "epoch": 3.0,
-      "eval_loss": 0.05235280096530914,
-      "eval_mse": 0.05235280389813637,
-      "eval_runtime": 10.3984,
-      "eval_samples_per_second": 961.689,
-      "eval_steps_per_second": 120.211,
-      "step": 9429
+      "eval_loss": 0.06556913256645203,
+      "eval_mse": 0.06556913494220615,
+      "eval_runtime": 13.2288,
+      "eval_samples_per_second": 755.924,
+      "eval_steps_per_second": 94.491,
+      "step": 6033
     },
     {
       "epoch": 3.0,
-      "step": 9429,
-      "total_flos": 9935679003367680.0,
-      "train_loss": 0.05866297316179509,
-      "train_runtime": 1207.603,
-      "train_samples_per_second": 499.597,
-      "train_steps_per_second": 7.808
+      "step": 6033,
+      "total_flos": 6357984788759040.0,
+      "train_loss": 0.07220485827706652,
+      "train_runtime": 846.782,
+      "train_samples_per_second": 455.926,
+      "train_steps_per_second": 7.125
     }
   ],
   "logging_steps": 500,
-  "max_steps": 9429,
+  "max_steps": 6033,
   "num_input_tokens_seen": 0,
   "num_train_epochs": 3,
   "save_steps": 500,
@@ -188,7 +146,7 @@
       "attributes": {}
     }
   },
-  "total_flos": 9935679003367680.0,
+  "total_flos": 6357984788759040.0,
   "train_batch_size": 64,
   "trial_name": null,
   "trial_params": null

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4a3b335a637ee1abd3e5da0d4f9e8ac74f5cd424a9c44c9efbe15e454133d934
+oid sha256:9c472a73a883ba5245b32b70e114642c495e951ce29acca84c258c8a402b2a81
 size 5368