autoevaluator
HF staff
Add evaluation results on the split config and test split of dair-ai/emotion
a6ef7b6
| language: | |
| - en | |
| license: apache-2.0 | |
| tags: | |
| - generated_from_trainer | |
| datasets: | |
| - dair-ai/emotion | |
| metrics: | |
| - accuracy | |
| - f1 | |
| base_model: distilbert-base-uncased | |
| model-index: | |
| - name: distilbert-base-uncased-finetuned-emotion | |
| results: | |
| - task: | |
| type: text-classification | |
| name: Text Classification | |
| dataset: | |
| name: emotion | |
| type: emotion | |
| config: split | |
| split: validation | |
| args: split | |
| metrics: | |
| - type: accuracy | |
| value: 0.9375 | |
| name: Accuracy | |
| - type: f1 | |
| value: 0.937890467332837 | |
| name: F1 | |
| - task: | |
| type: text-classification | |
| name: Text Classification | |
| dataset: | |
| name: dair-ai/emotion | |
| type: dair-ai/emotion | |
| config: split | |
| split: test | |
| metrics: | |
| - type: accuracy | |
| value: 0.93 | |
| name: Accuracy | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNTExYzEzMWNmYTNlYmI0NWNjYTIwMzU3MmUyYmY0ZDZjMjQwOTMzYWMwOTZiY2U4YTA2ZDE0NmM2YzNlMzNkZiIsInZlcnNpb24iOjF9.rwu31KKjXkNu7uVA-vxi4NX8Fd2cJrnAmWbIIt174dmi24nlB56g7IDBfTrGzFdnMzkCuDpLng8pnvXFoN3ZCg | |
| - type: f1 | |
| value: 0.8869023464973423 | |
| name: F1 Macro | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzI0NjEzNmEwM2VjNTg0MjY2ZmYwZTA0ZGJkOTI2ZWFlNTUxNzA1ZGNkNzNhZGQ1NGZlZTVhZGY4ZGUwZjc5YyIsInZlcnNpb24iOjF9.PLxM2vSrYDzbdKVaK3QqI_J8ujKvTUfpSfQmC-MsHNgTw7329UaiROWhe1bhadQcgNolLgtwlLFhXyR593fGAA | |
| - type: f1 | |
| value: 0.93 | |
| name: F1 Micro | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDFlMGY1ZDljZTQ3NzI5N2FmNTFkOWY0N2EzNTk5MGZhOWM4MDVkYjQ2NDk1NTU3MWZiMDBhNTc4YWE2MTFkOSIsInZlcnNpb24iOjF9.5wCoKKKKl0p9S0nAN2OuiPe3c9VnBmTHnJjWWdHgBmcbJ2CrVjZzejUXnfpsuaVJSxSmOZfdI6h18z_fQRgqAQ | |
| - type: f1 | |
| value: 0.9300315549555708 | |
| name: F1 Weighted | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZjcxNGMyNWE5ODg3MDczMzM5ZjE0YmU3YTRmNTM4MTQwYjhmMDcxOGU1NGU4YTBmZGI5NmM5OTRiY2VhYzQ3ZCIsInZlcnNpb24iOjF9.l1MbXmlI8txam4EttXSOWaIgfN9sKe0ZBKc_TXwWre8DNgPFVwVD4jWeQxlMRC0LtWIL5fIqEdv8qj5DqVz1BQ | |
| - type: precision | |
| value: 0.892405250997362 | |
| name: Precision Macro | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjg4Y2U3ODNkZjRkMWQ1YzZjMWUzYWRkOTk1YzJkNmU1NmVkZGNjZDA5NmQzNmVkNDExNzBlZDZiZDg1YWU2YSIsInZlcnNpb24iOjF9.WoctTcjRNIbgZo4pqUQVbKmOc0iudqmqO9ABr043llfHLIWe8ZjMyFek3OBUTgw7wVe35UDNGeRc308OxS7CAQ | |
| - type: precision | |
| value: 0.93 | |
| name: Precision Micro | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOTczZGUwMTQ1MWY2NGE4YWU0NWU3YjA0MzdhMGUyYWUzYjAxOTgzYmYwNmNhYWIxZTBhODE4YjMyOTc5NDAwYSIsInZlcnNpb24iOjF9.Iy2em6yVS3K4izKHRhnap2RWHgZQ5hup8nmtNVmb7avz5x3HWUnzwAUq_EsWht_7Hf59YUPuWW_xv9EZXZYVDQ | |
| - type: precision | |
| value: 0.9314726605632766 | |
| name: Precision Weighted | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMDljOTNmMjZjNWFkZGQzNTIxNzZhNTRjYTEwN2E0MjdmYWY2MzdjMmUyMzZlZGQzZmFlMDJiNmIxMDUxYTM1MyIsInZlcnNpb24iOjF9.S2ERnUMdG3WZyRIpYY4ZOPPMdy3VvUVcNA3sQ8uij5S-1upDs5DeSxqgXKGggxANkj1lSOda8sDSLYgswTV4DQ | |
| - type: recall | |
| value: 0.8858832612260938 | |
| name: Recall Macro | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYjllMGU4MmRiYjljNDQ0NTEwOGIzOTMzMDI1NThjZDdhYjc0OTNmYTcwODU0ZTNmYjEwOWQ4Mjg0MDAyNjNkOSIsInZlcnNpb24iOjF9.yl3TFQCdTFGGdxbDdTFK4yAY8segyexnvqY0uViW9bEQsfXVgP-lE3veKnsakAKo4HD6nlthHUxtgJLZhYcfAg | |
| - type: recall | |
| value: 0.93 | |
| name: Recall Micro | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYWZkNDg2YjdiZTMzMDljNWNiMzk0MzQ2MjA0NzBjMzkwMWE3OWI1YTgwNTkxODhjYzFkMjQ2Yjg4ZjkyZDE1ZCIsInZlcnNpb24iOjF9.I76FPjzyPTVxC8w5ZgDOnsGOx1weTzizDujT98WPGO3AnibtlJJJhPeQdBKxe80LGE7QpjLx0e-R9UvjpnJmAQ | |
| - type: recall | |
| value: 0.93 | |
| name: Recall Weighted | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOGY2ZmI1OWU3YWVjMDM3NjZjNjhlNWZiMDhlMzA3MDU0ZjExYTA5OTA1ZDBjY2YwYTA0NDE2ODY1ZGIwYmU2OCIsInZlcnNpb24iOjF9.9NACQL-gIUD57HmB62GIo5nqJJVy9k_iUo72onHCyfpCa9K0XKhi_UoKJNoV5htumHD2zKKucwUx1p8X17bUAg | |
| - type: loss | |
| value: 0.1600879579782486 | |
| name: loss | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMTYzNzkwMWY1ZDMwMTVhN2Y4NDkwNTgyYWNjNjkwMTIwYTNjZjY0N2M0MWM5NjNlNjA3ZWJmY2VhNTQ2MGUyOCIsInZlcnNpb24iOjF9.YYmdyZk5D8B0Fb7M2ysyLBGdSVxGOSgWGfXhz7h0UZaeLijo04IiFQfNWiYNtd_UZ2QcpMeZtkuWdMK8ZHPxBQ | |
| # distilbert-base-uncased-finetuned-emotion | |
| This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the emotion dataset. | |
| It achieves the following results on the evaluation set: | |
| - Loss: 0.1448 | |
| - Accuracy: 0.9375 | |
| - F1: 0.9379 | |
| The notebook used to fine-tune this model may be found [HERE](https://www.kaggle.com/marcoloureno/distilbert-base-uncased-finetuned-emotion). | |
| ## Model description | |
| DistilBERT is a transformers model, smaller and faster than BERT, which was pretrained on the same corpus in a | |
| self-supervised fashion, using the BERT base model as a teacher. This means it was pretrained on the raw texts only, | |
| with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic | |
| process to generate inputs and labels from those texts using the BERT base model. More precisely, it was pretrained | |
| with three objectives: | |
| - Distillation loss: the model was trained to return the same probabilities as the BERT base model. | |
| - Masked language modeling (MLM): this is part of the original training loss of the BERT base model. When taking a | |
| sentence, the model randomly masks 15% of the words in the input then run the entire masked sentence through the | |
| model and has to predict the masked words. This is different from traditional recurrent neural networks (RNNs) that | |
| usually see the words one after the other, or from autoregressive models like GPT which internally mask the future | |
| tokens. It allows the model to learn a bidirectional representation of the sentence. | |
| - Cosine embedding loss: the model was also trained to generate hidden states as close as possible as the BERT base | |
| model. | |
| This way, the model learns the same inner representation of the English language than its teacher model, while being | |
| faster for inference or downstream tasks. | |
| ## Intended uses & limitations | |
| [Emotion](https://huggingface.co/datasets/dair-ai/emotion) is a dataset of English Twitter messages with six basic emotions: anger, fear, joy, love, sadness, and surprise. This dataset was developed for the paper entitled "CARER: Contextualized Affect Representations for Emotion Recognition" (Saravia et al.) through noisy labels, annotated via distant | |
| supervision as in the paper"Twitter sentiment classification using distant supervision" (Go et al). | |
| The DistilBERT model was fine-tuned to this dataset, allowing for the classification of sentences into one of the six basic emotions (anger, fear, joy, love, sadness, and surprise). | |
| ## Training procedure | |
| ### Training hyperparameters | |
| The following hyperparameters were used during training: | |
| - learning_rate: 5e-05 | |
| - train_batch_size: 64 | |
| - eval_batch_size: 64 | |
| - seed: 42 | |
| - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 | |
| - lr_scheduler_type: linear | |
| - num_epochs: 2 | |
| ### Training results | |
| | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 | | |
| |:-------------:|:-----:|:----:|:---------------:|:--------:|:------:| | |
| | 0.5337 | 1.0 | 250 | 0.1992 | 0.927 | 0.9262 | | |
| | 0.1405 | 2.0 | 500 | 0.1448 | 0.9375 | 0.9379 | | |
| ### Framework versions | |
| - Transformers 4.30.2 | |
| - Pytorch 2.0.0 | |
| - Datasets 2.1.0 | |
| - Tokenizers 0.13.3 |