|
--- |
|
title: FairEval |
|
tags: |
|
- evaluate |
|
- metric |
|
description: "Fair Evaluation for Squence labeling" |
|
sdk: gradio |
|
sdk_version: 3.0.2 |
|
app_file: app.py |
|
pinned: false |
|
--- |
|
|
|
# Fair Evaluation for Sequence Labeling |
|
|
|
## Metric Description |
|
The traditional evaluation of NLP labeled spans with precision, recall, and F1-score leads to double penalties for |
|
close-to-correct annotations. As [Manning (2006)](https://nlpers.blogspot.com/2006/08/doing-named-entity-recognition-dont.html) |
|
argues in an article about named entity recognition, this can lead to undesirable effects when systems are optimized for these traditional metrics. |
|
To address these issues, this metric provides an implementation of FairEval, proposed by [Ortmann (2022)](https://aclanthology.org/2022.lrec-1.150.pdf). |
|
|
|
## How to Use |
|
FairEval outputs the error count (TP, FP, etc.) and resulting scores (Precision, Recall and F1) from a reference list of |
|
spans compared against a predicted one. The user can choose to see traditional or fair error counts and scores by |
|
switching the argument **mode**. |
|
|
|
The user can also choose to see the metric parameters (TP, FP...) as absolute count, as a percentage with respect to the |
|
total number of errors or with respect to the total number of ground truth entities through the argument **error_format**. |
|
|
|
The minimal example is: |
|
|
|
```python |
|
faireval = evaluate.load("hpi-dhc/FairEval") |
|
pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER', 'O']] |
|
ref = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER', 'O']] |
|
results = faireval.compute(predictions=pred, references=ref) |
|
``` |
|
|
|
### Inputs |
|
FairEval handles input annotations as seqeval. The supported formats are IOB1, IOB2, IOE1, IOE2 and IOBES. |
|
Predicted sentences must have the same number of tokens as the references. |
|
- **predictions** *(list)*: a list of lists of predicted labels, i.e. estimated targets as returned by a tagger. |
|
- **references** *(list)*: list of ground truth reference labels. |
|
|
|
The optional arguments are: |
|
- **mode** *(str)*: 'fair', 'traditional' ot 'weighted. Controls the desired output. The default value is 'fair'. |
|
- 'traditional': equivalent to seqeval's 'strict' mode. Bear in mind that the default mode for seqeval is 'relaxed', which does not match with any of faireval modes. |
|
- 'fair': default fair score calculation. Fair will also show traditional scores for comparison. |
|
- 'weighted': custom score calculation with the weights passed. Weighted will also show traditional scores for comparison. |
|
- **weights** *(dict)*: dictionary with the weight of each error for the custom score calculation. |
|
- **error_format** *(str)*: 'count', 'error_ratio' or 'entity_ratio'. Controls the desired output for TP, FP, BE, LE, etc. Default value is 'count'. |
|
- 'count': absolute count of each parameter. |
|
- 'error_ratio': precentage with respect to the total errors that each parameter represents. |
|
- 'entity_ratio': precentage with respect to the total number of ground truth entites that each parameter represents. |
|
- **zero_division** *(str)*: which value to substitute as a metric value when encountering zero division. Should be one of [0,1,"warn"]. "warn" acts as 0, but the warning is raised. |
|
- **suffix** *(boolean)*: True if the IOB tag is a suffix (after type) instead of a prefix (before type), False otherwise. The default value is False, i.e. the IOB tag is a prefix (before type). |
|
- **scheme** *(str)*: the target tagging scheme, which can be one of [IOB1, IOB2, IOE1, IOE2, IOBES, BILOU]. The default value is None. |
|
|
|
### Output Values |
|
A dictionary with: |
|
- Overall error parameter count (or ratio) and resulting scores. |
|
- A nested dictionary per label with its respective error parameter count (or ratio) and resulting scores |
|
|
|
If mode is 'traditional', the error parameters shown are the classical TP, FP and FN. If mode is 'fair' or 'weighted', |
|
TP remain the same, FP and FN are shown as per the fair definition and additional errors BE, LE and LBE are shown. |
|
|
|
### Examples |
|
Considering the following input annotated sentences: |
|
```python |
|
>>> r1 = ['O', 'O', 'B-PER', 'I-PER', 'O', 'B-PER'] |
|
>>> p1 = ['O', 'O', 'B-PER', 'I-PER', 'O', 'O' ] #1FN |
|
>>> |
|
>>> r2 = ['O', 'B-INT', 'B-OUT'] |
|
>>> p2 = ['B-INT', 'I-INT', 'B-OUT'] #1BE |
|
>>> |
|
>>> r3 = ['B-INT', 'I-INT', 'B-OUT'] |
|
>>> p3 = ['B-OUT', 'O', 'B-PER'] #1LBE, 1LE |
|
>>> |
|
>>> y_true = [r1, r2, r3] |
|
>>> y_pred = [p1, p2, p3] |
|
``` |
|
|
|
The output for different modes and error_formats is: |
|
```python |
|
>>> faireval.compute(predictions=y_pred, references=y_true, mode='fair', error_format='count') |
|
{"PER": {"precision": 1.0, "recall": 0.5, "f1": 0.6666, |
|
"trad_prec": 0.5, "trad_rec": 0.5, "trad_f1": 0.5, |
|
"TP": 1, "FP": 0.0, "FN": 1.0, "LE": 0.0, "BE": 0.0, "LBE": 0.0}, |
|
"INT": {"precision": 0.0, "recall": 0.0, "f1": 0.0, |
|
"trad_prec": 0.0, "trad_rec": 0.0, "trad_f1": 0.0, |
|
"TP": 0, "FP": 0.0, "FN": 0.0, "LE": 0.0, "BE": 1.0, "LBE": 1.0}, |
|
"OUT": {"precision": 0.6666, "recall": 0.6666, "f1": 0.666, |
|
"trad_prec": 0.5, "trad_rec": 0.5, "trad_f1": 0.5, |
|
"TP": 1, "FP": 0.0, "FN": 0.0, "LE": 1.0, "BE": 0.0, "LBE": 0.0}, |
|
"overall_precision": 0.5714, "overall_recall": 0.4444, "overall_f1": 0.5, |
|
"overall_trad_prec": 0.4, "overall_trad_rec": 0.3333, "overall_trad_f1": 0.3636, |
|
"TP": 2, "FP": 0.0, "FN": 1.0, "LE": 1.0, "BE": 1.0, "LBE": 1.0} |
|
``` |
|
|
|
```python |
|
>>> faireval.compute(predictions=y_pred, references=y_true, mode='traditional', error_format='count') |
|
{"PER": {"precision": 0.5, "recall": 0.5, "f1": 0.5, |
|
"TP": 1, "FP": 1.0, "FN": 1.0}, |
|
"INT": {"precision": 0.0, "recall": 0.0, "f1": 0.0, |
|
"TP": 0, "FP": 1.0, "FN": 2.0}, |
|
"OUT": {"precision": 0.5, "recall": 0.5, "f1": 0.5, |
|
"TP": 1, "FP": 1.0, "FN": 1.0}, |
|
"overall_precision": 0.4, "overall_recall": 0.3333, "overall_f1": 0.3636, |
|
"TP": 2, "FP": 3.0, "FN": 4.0} |
|
``` |
|
|
|
```python |
|
>>> faireval.compute(predictions=y_pred, references=y_true, mode='traditional', error_format='error_ratio') |
|
{"PER": {"precision": 0.5, "recall": 0.5, "f1": 0.5, |
|
"TP": 1, "FP": 0.1428, "FN": 0.1428}, |
|
"INT": {"precision": 0.0, "recall": 0.0, "f1": 0.0, |
|
"TP": 0, "FP": 0.1428, "FN": 0.2857}, |
|
"OUT": {"precision": 0.5, "recall": 0.5, "f1": 0.5, |
|
"TP": 1, "FP": 0.1428, "FN": 0.1428}, |
|
"overall_precision": 0.4, "overall_recall": 0.3333, "overall_f1": 0.3636, |
|
"TP": 2, "FP": 0.4285, "FN": 0.5714} |
|
``` |
|
|
|
### Values from Popular Papers |
|
|
|
#### CoNLL2003 |
|
Computing the evaluation metrics on the results from [this model](https://huggingface.co/elastic/distilbert-base-uncased-finetuned-conll03-english) |
|
run on the test split of [CoNLL2003 dataset](https://huggingface.co/datasets/conll2003), we obtain the following F1-Scores: |
|
|
|
| F1 Scores | overall | location | miscellaneous | organization | person | |
|
|-----------------|--------:|---------:|-------------:|-------------:|-------:| |
|
| fair | 0.94 | 0.96 | 0.85 | 0.92 | 0.97 | |
|
| traditional | 0.90 | 0.92 | 0.79 | 0.87 | 0.96 | |
|
| seqeval strict | 0.90 | 0.92 | 0.79 | 0.87 | 0.96 | |
|
| seqeval relaxed | 0.90 | 0.92 | 0.78 | 0.87 | 0.96 | |
|
|
|
With error count: |
|
|
|
| | overall (trad) | overall (fair) | location (trad)| location (fair) | miscellaneous (trad)| miscellaneous (fair) | organization (trad)| organization (fair) | person (trad)| person (fair) | |
|
|-----|--------:|-----:|---------:|-----:|-------------:|----:|-------------:|-----:|-------:|-----:| |
|
| TP | 5104 | 5104 | 1545 | 1545 | 561 | 561 | 1452 | 1452 | 1546 | 1546 | |
|
| FP | 534 | 126 | 128 | 20 | 154 | 48 | 208 | 47 | 44 | 11 | |
|
| FN | 544 | 124 | 123 | 13 | 141 | 47 | 209 | 47 | 71 | 17 | |
|
| LE | | 219 | | 62 | | 41 | | 73 | | 43 | |
|
| BE | | 126 | | 16 | | 46 | | 53 | | 11 | |
|
| LBE | | 87 | | 32 | | 13 | | 41 | | 1 | |
|
|
|
#### WNUT-17 |
|
Computing the evaluation metrics on the results from [this model](https://huggingface.co/muhtasham/bert-small-finetuned-wnut17-ner) |
|
run on the test split of [WNUT-17 dataset](https://huggingface.co/datasets/wnut_17), we obtain the following F1-Scores: |
|
|
|
| | overall | location | group | person | creative work | corporation | product | |
|
|-----------------|--------:|---------:|-------:|-------:|--------------:|------------:|--------:| |
|
| fair | 0.37 | 0.58 | 0.02 | 0.58 | 0.0 | 0.03 | 0.0 | |
|
| traditional | 0.35 | 0.53 | 0.02 | 0.55 | 0.0 | 0.02 | 0.0 | |
|
| seqeval strict | 0.35 | 0.53 | 0.02 | 0.55 | 0.0 | 0.02 | 0.0 | |
|
| seqeval relaxed | 0.34 | 0.49 | 0.02 | 0.55 | 0.0 | 0.02 | 0.0 | |
|
|
|
With error count: |
|
|
|
| | overall (trad)| overall (fair) | location (trad)| location (fair) | group (trad)| group (fair) | person (trad)| person (fair) | creative work (trad)| creative work (fair) | corporation (trad)| corporation (fair) | product (trad)| product (fair) | |
|
|-----|--------:|----:|---------:|---:|------:|----:|-------:|----:|--------------:|----:|------------:|---:|--------:|----:| |
|
| TP | 255 | 255 | 67 | 67 | 2 | 2 | 185 | 185 | 0 | 0 | 1 | 1 | 0 | 0 | |
|
| FP | 135 | 31 | 38 | 10 | 20 | 3 | 60 | 16 | 0 | 0 | 17 | 2 | 0 | 0 | |
|
| FN | 824 | 725 | 83 | 71 | 163 | 135 | 244 | 233 | 142 | 120 | 65 | 54 | 127 | 112 | |
|
| LE | | 47 | | 4 | | 18 | | 2 | | 6 | | 7 | | 10 | |
|
| BE | | 30 | | 10 | | 4 | | 13 | | 0 | | 3 | | 0 | |
|
| LBE | | 29 | | 1 | | 6 | | 0 | | 16 | | 1 | | 5 | |
|
|
|
## Limitations and Bias |
|
The metric is restricted to the input schemes admitted by seqeval. For example, the application does not support numerical |
|
label inputs (odd for Beginning, even for Inside and zero for Outside). |
|
|
|
The choice of custom weights for wheighted evaluation is subjective to the user. Neither weighted nor fair evaluations |
|
can be compared to traditional span-based metrics used in other pairs of datasets-models. |
|
|
|
## Citation |
|
Ortmann, Katrin. 2022. Fine-Grained Error Analysis and Fair Evaluation of Labeled Spans. In *Proceedings of the Language Resources and Evaluation Conference (LREC)*, Marseille, France, pages 1400–1407. [PDF](https://aclanthology.org/2022.lrec-1.150.pdf) |
|
|
|
```bibtex |
|
@inproceedings{ortmann2022, |
|
title = {Fine-Grained Error Analysis and Fair Evaluation of Labeled Spans}, |
|
author = {Katrin Ortmann}, |
|
url = {https://aclanthology.org/2022.lrec-1.150}, |
|
year = {2022}, |
|
date = {2022-06-21}, |
|
booktitle = {Proceedings of the Language Resources and Evaluation Conference (LREC)}, |
|
pages = {1400-1407}, |
|
publisher = {European Language Resources Association}, |
|
address = {Marseille, France}, |
|
pubstate = {published}, |
|
type = {inproceedings} |
|
} |
|
``` |
