Hugging Face's logo

gauneg
/
deberta-v3-base-absa-ate-sentiment

Token Classification
Transformers
Safetensors
English
deberta-v2
Inference Endpoints
Model card Files Community
gauneg commited on
Commit
d080b97
·
verified ·
1 Parent(s): 8c30c6f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +179 -1
README.md CHANGED
@@ -6,4 +6,182 @@ base_model:
6
  - microsoft/deberta-v3-base
7
  pipeline_tag: token-classification
8
  library_name: transformers
9
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
  - microsoft/deberta-v3-base
7
  pipeline_tag: token-classification
8
  library_name: transformers
9
+ ---
10
+
11
+
12
+ # Training
13
+ This model is designed for token classification tasks, enabling it to extract aspect terms and predict the sentiment polarity associated with the extracted aspect terms.
14
+ The extracted aspect terms will be the span(s) from the input text on which a sentiment is being expressed.
15
+
16
+ ## Datasets
17
+ This model has been trained on the following datasets:
18
+
19
+ 1. Aspect Based Sentiment Analysis SemEval Shared Tasks ([2014](https://aclanthology.org/S14-2004/), [2015](https://aclanthology.org/S15-2082/), [2016](https://aclanthology.org/S16-1002/))
20
+ 2. Multi-Aspect Multi-Sentiment [MAMS](https://aclanthology.org/D19-1654/)
21
+
22
+ # Use
23
+
24
+ * Making token level inferences with Auto classes
25
+
26
+ ```python
27
+ from transformers import AutoTokenizer, AutoModelForTokenClassification
28
+ model_id = "gauneg/deberta-v3-base-absa-ate-sentiment"
29
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
30
+
31
+
32
+
33
+ # the sequence of labels used during training
34
+ labels = {"B-neu": 1, "I-neu": 2, "O": 0, "B-neg": 3, "B-con": 4, "I-pos": 5, "B-pos": 6, "I-con": 7, "I-neg": 8, "X": -100}
35
+ id2lab = {idx: lab for lab, idx in labels.items()}
36
+ lab2id = {lab: idx for lab, idx in labels.items()}
37
+
38
+ model = AutoModelForTokenClassification.from_pretrained("../models/deberta-v3-base-bio-w-pol/",
39
+ num_labels=len(labels), id2label=id2lab, label2id=lab2id)
40
+
41
+ # making one prediction at a time (should be padded/batched and truncated for efficiency)
42
+ text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
43
+ tok_inputs = tokenizer(text_input, return_tensors="pt")
44
+
45
+
46
+ y_pred = model(**tok_inputs) # predicting the logits
47
+
48
+
49
+ # selecting the most favoured labels for each token from the logits
50
+ y_pred_fin = y_pred.logits.argmax(dim=-1)[0]
51
+
52
+
53
+ # since first and the last tokens are excluded ([CLS] and [SEP]) they have to be removed before decoding
54
+ decoded_pred = [id2lab[logx.item()] for logx in y_pred_fin[1:-1]]
55
+
56
+ ## displaying the input tokens with predictions and skipping [CLS] and [SEP] tokens at the beginning and the end respectively
57
+ decoded_toks = tok_inputs['input_ids'][0][1:-1]
58
+ tok_levl_pred = list(zip(tokenizer.convert_ids_to_tokens(decoded_toks), decoded_pred))
59
+ ```
60
+ Expected output
61
+
62
+ ```bash
63
+ [('▁Been', 'O'),
64
+ ('▁here', 'O'),
65
+ ('▁a', 'O'),
66
+ ('▁few', 'O'),
67
+ ('▁times', 'O'),
68
+ ('▁and', 'O'),
69
+ ('▁food', 'B-pos'),
70
+ ('▁has', 'O'),
71
+ ('▁always', 'O'),
72
+ ('▁been', 'O'),
73
+ ('▁good', 'O'),
74
+ ('▁but', 'O'),
75
+ ('▁service', 'B-neg'),
76
+ ('▁really', 'O'),
77
+ ('▁suffers', 'O'),
78
+ ('▁when', 'O'),
79
+ ('▁it', 'O'),
80
+ ('▁gets', 'O'),
81
+ ('▁crowded', 'O'),
82
+ ('.', 'O')]
83
+ ```
84
+
85
+ * Making end-to-end inference with a pipeline
86
+
87
+ ```python
88
+
89
+ from transformers import pipeline
90
+
91
+ ate_sent_pipeline = pipeline(task='ner',
92
+ aggregation_strategy='simple',
93
+ model="gauneg/deberta-v3-base-absa-ate-sentiment")
94
+
95
+ text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
96
+ ate_sent_pipeline(text_input)
97
+
98
+ ```
99
+ Expected output
100
+
101
+ ```bash
102
+ [{'entity_group': 'pos', #sentiment polarity
103
+ 'score': 0.87505656,
104
+ 'word': 'food', # aspect term
105
+ 'start': 25,
106
+ 'end': 30},
107
+ {'entity_group': 'neg',# sentiment polarity
108
+ 'score': 0.4558051,
109
+ 'word': 'service', #aspect term
110
+ 'start': 55,
111
+ 'end': 63}]
112
+
113
+ ```
114
+
115
+
116
+ # Evaluation on Benchmark Test Datasets
117
+
118
+ The first evaluation is for token-extraction task without considering the polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens
119
+ on which the sentiments have been expressed. (scores are expressed as micro-averages of B-I-O labels)
120
+
121
+ # ATE (Aspect Term Extraction Only)
122
+ | Test Dataset | Base Model | Fine-tuned Model | Precision | Recall | F1 Score |
123
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
124
+ |hotel reviews (SemEval 2015)|(this) microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|71.16|73.92|71.6|
125
+ |hotel reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|70.92|72.28|71.07|
126
+ |hotel reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|64.05|79.69|70.0|
127
+ |hotel reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|66.29|72.78|68.92|
128
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
129
+ |laptop reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|70.58|61.52|64.21|
130
+ |laptop reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|66.38|50.62|54.31|
131
+ |laptop reviews (SemEval 2014)|(this) microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|70.82|48.97|52.08|
132
+ |laptop reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|73.61|46.38|49.87|
133
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
134
+ |MAMS-ATE (2019)|(this) microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|81.07|79.66|80.35|
135
+ |MAMS-ATE (2019)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|79.91|78.95|79.39|
136
+ |MAMS-ATE (2019)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|74.46|84.5|78.75|
137
+ |MAMS-ATE (2019)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|77.8|79.81|78.75|
138
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
139
+ |restaurant reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|88.59|87.0|87.45|
140
+ |restaurant reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|92.26|82.95|86.57|
141
+ |restaurant reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|93.07|81.95|86.32|
142
+ |restaurant reviews (SemEval 2014)|(this) microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|92.94|81.71|86.01|
143
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
144
+ |restaurant reviews (SemEval 2015)|(this) microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|72.91|75.4|72.74|
145
+ |restaurant reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|70.54|77.48|72.63|
146
+ |restaurant reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|68.32|79.84|72.28|
147
+ |restaurant reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|71.94|74.75|71.84|
148
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
149
+ |restaurant reviews (SemEval 2016)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|70.22|75.83|71.84|
150
+ |restaurant reviews (SemEval 2016)|(this) microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|71.54|73.38|71.2|
151
+ |restaurant reviews (SemEval 2016)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|71.35|72.78|70.85|
152
+ |restaurant reviews (SemEval 2016)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|66.68|77.97|70.79|
153
+
154
+ # Aspect Sentiment Evaluation
155
+ This evaluation considers token-extraction task with polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens
156
+ on which the sentiments have been expressed along with the polarity of the sentiments. (scores are expressed as macro-averages)
157
+ | Test Dataset | Base Model | Fine-tuned Model | Precision | Recall | F1 Score |
158
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
159
+ |hotel reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|51.92|65.55|54.94|
160
+ |hotel reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|54.62|53.65|54.08|
161
+ |hotel reviews (SemEval 2015)|(this) microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|55.43|56.53|54.03|
162
+ |hotel reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|52.88|55.19|53.85|
163
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
164
+ |laptop reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|44.25|41.55|42.81|
165
+ |laptop reviews (SemEval 2014)|(this) microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|46.15|33.23|37.09|
166
+ |laptop reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|41.7|34.38|36.93|
167
+ |laptop reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|44.98|31.87|35.67|
168
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
169
+ |MAMS-ATE (2019)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|72.06|72.98|72.49|
170
+ |MAMS-ATE (2019)|(this) microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|72.97|71.63|72.26|
171
+ |MAMS-ATE (2019)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|69.34|73.3|71.07|
172
+ |MAMS-ATE (2019)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|65.74|75.11|69.77|
173
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
174
+ |restaurant reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|61.15|58.46|59.74|
175
+ |restaurant reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|60.13|56.81|58.13|
176
+ |restaurant reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|56.79|59.3|57.93|
177
+ |restaurant reviews (SemEval 2014)|(this) microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|58.99|54.76|56.45|
178
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
179
+ |restaurant reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|53.89|55.7|54.11|
180
+ |restaurant reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|54.36|55.38|53.6|
181
+ |restaurant reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|51.67|56.58|53.29|
182
+ |restaurant reviews (SemEval 2015)|(this) microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|54.55|53.68|53.12|
183
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
184
+ |restaurant reviews (SemEval 2016)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|53.7|60.49|55.05|
185
+ |restaurant reviews (SemEval 2016)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|52.31|54.58|52.33|
186
+ |restaurant reviews (SemEval 2016)|(this) microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|52.07|54.58|52.15|
187
+ |restaurant reviews (SemEval 2016)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|49.07|56.5|51.25|