Edit model card

Overview

This is a multi-label, multi-class linear classifer for emotions that works with sentence-transformers/all-MiniLM-L6-v2, having been trained on the go_emotions dataset.

Labels

The 28 labels from the go_emotions dataset are:

['admiration', 'amusement', 'anger', 'annoyance', 'approval', 'caring', 'confusion', 'curiosity', 'desire', 'disappointment', 'disapproval', 'disgust', 'embarrassment', 'excitement', 'fear', 'gratitude', 'grief', 'joy', 'love', 'nervousness', 'optimism', 'pride', 'realization', 'relief', 'remorse', 'sadness', 'surprise', 'neutral']

Metrics (exact match of labels per item)

This is a multi-label, multi-class dataset, so each label is effectively a separate binary classification. Evaluating across all labels per item in the go_emotions test split the metrics are shown below.

Optimising the threshold per label to optimise the F1 metric, the metrics (evaluated on the go_emotions test split) are:

  • Precision: 0.384
  • Recall: 0.438
  • F1: 0.397

Weighted by the relative support of each label in the dataset, this is:

  • Precision: 0.443
  • Recall: 0.552
  • F1: 0.484

Using a fixed threshold of 0.5 to convert the scores to binary predictions for each label, the metrics (evaluated on the go_emotions test split, and unweighted by support) are:

  • Precision: 0.551
  • Recall: 0.211
  • F1: 0.261

Metrics (per-label)

This is a multi-label, multi-class dataset, so each label is effectively a separate binary classification and metrics are better measured per label.

Optimising the threshold per label to optimise the F1 metric, the metrics (evaluated on the go_emotions test split) are:

f1 precision recall support threshold
admiration 0.529 0.499 0.563 504 0.25
amusement 0.733 0.672 0.807 264 0.20
anger 0.394 0.363 0.429 198 0.15
annoyance 0.293 0.252 0.350 320 0.15
approval 0.292 0.345 0.254 351 0.20
caring 0.320 0.270 0.393 135 0.15
confusion 0.291 0.276 0.307 153 0.15
curiosity 0.366 0.307 0.454 284 0.15
desire 0.317 0.269 0.386 83 0.15
disappointment 0.159 0.127 0.212 151 0.10
disapproval 0.306 0.341 0.277 267 0.20
disgust 0.405 0.412 0.398 123 0.20
embarrassment 0.364 0.414 0.324 37 0.35
excitement 0.296 0.232 0.408 103 0.15
fear 0.496 0.576 0.436 78 0.40
gratitude 0.793 0.787 0.798 352 0.30
grief 0.323 0.200 0.833 6 0.45
joy 0.402 0.341 0.491 161 0.15
love 0.640 0.679 0.605 238 0.30
nervousness 0.263 0.333 0.217 23 0.70
optimism 0.433 0.453 0.414 186 0.20
pride 0.429 0.500 0.375 16 0.50
realization 0.177 0.159 0.200 145 0.10
relief 0.182 0.182 0.182 11 0.40
remorse 0.541 0.500 0.589 56 0.30
sadness 0.461 0.467 0.455 156 0.20
surprise 0.302 0.299 0.305 141 0.15
neutral 0.620 0.505 0.803 1787 0.30

The thesholds are stored in thresholds.json.

Use with ONNXRuntime

The input to the model is called logits, and there is one output per label. Each output produces a 2d array, with 1 row per input row, and each row having 2 columns - the first being a proba output for the negative case, and the second being a proba output for the positive case.

# Assuming you have embeddings from all-MiniLM-L6-v2 for the input sentences
# E.g. produced from sentence-transformers such as:
#      huggingface.co/sentence-transformers/all-MiniLM-L6-v2
#      or from an ONNX version E.g. huggingface.co/Xenova/all-MiniLM-L6-v2

print(embeddings.shape)  # E.g. a batch of 1 sentence
> (1, 384)

import onnxruntime as ort

sess = ort.InferenceSession("path_to_model_dot_onnx", providers=['CPUExecutionProvider'])

outputs = [o.name for o in sess.get_outputs()]  # list of labels, in the order of the outputs
preds_onnx = sess.run(_outputs, {'logits': embeddings})
# preds_onnx is a list with 28 entries, one per label,
# each with a numpy array of shape (1, 2) given the input was a batch of 1

print(outputs[0])
> surprise
print(preds_onnx[0])
> array([[0.97136074, 0.02863926]], dtype=float32)

# load thresholds.json and use that (per label) to convert the positive case score to a binary prediction 

Commentary on the dataset

Some labels (E.g. gratitude) when considered independently perform very strongly, whilst others (E.g. relief) perform very poorly.

This is a challenging dataset. Labels such as relief do have much fewer examples in the training data (less than 100 out of the 40k+, and only 11 in the test split).

But there is also some ambiguity and/or labelling errors visible in the training data of go_emotions that is suspected to constrain the performance. Data cleaning on the dataset to reduce some of the mistakes, ambiguity, conflicts and duplication in the labelling would produce a higher performing model.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Examples
Inference API (serverless) has been turned off for this model.

Dataset used to train SamLowe/all-MiniLM-L6-v2-go_emotions-classifier-onnx