---
title: FairEval
tags:
- evaluate
- metric
description: "TODO: add a description here"
sdk: gradio
sdk_version: 3.0.2
app_file: app.py
pinned: false
---

# Fair Evaluation for Sequence Labeling

## Metric Description
The traditional evaluation of NLP labeled spans with precision, recall, and F1-score leads to double penalties for 
close-to-correct annotations. As [Manning (2006)](https://nlpers.blogspot.com/2006/08/doing-named-entity-recognition-dont.html) 
argues in an article about named entity recognition, this can lead to undesirable effects when systems are optimized for these traditional metrics.
To address these issues, this metric provides an implementation of FairEval, proposed by [Ortmann (2022)](https://aclanthology.org/2022.lrec-1.150.pdf).

## How to Use
FairEval outputs the error count (TP, FP, etc.) and resulting scores (Precision, Recall and F1) from a reference list of 
spans compared against a predicted one. The user can choose to see traditional or fair error counts and scores by 
switching the argument **mode**. 

The user can also choose to see the metric parameters (TP, FP...) as absolute count, as a percentage with respect to the 
total number of errors or with respect to the total number of ground truth entities through the argument **error_format**.

The minimal example is:

```python
faireval = evaluate.load("hpi-dhc/FairEval")
pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER', 'O']]
ref =  [['O', 'O', 'O',      'B-MISC', 'I-MISC', 'I-MISC', 'O', 'B-PER', 'I-PER', 'O']]
results = faireval.compute(predictions=pred, references=ref)
```

### Inputs
FairEval handles input annotations as seqeval. The supported formats are IOB1, IOB2, IOE1, IOE2 and IOBES. 
Predicted sentences must have the same number of tokens as the references.
- **predictions** *(list)*: a list of lists of predicted labels, i.e. estimated targets as returned by a tagger.
- **references** *(list)*: list of ground truth reference labels. 

The optional arguments are:
- **mode** *(str)*: 'fair', 'traditional' ot 'weighted. Controls the desired output. The default value is 'fair'.
  - 'traditional': equivalent to seqeval's metrics / classic span-based evaluation.
  - 'fair': default fair score calculation.
  - 'weighted': custom score calculation with the weights passed. 
- **weights** *(dict)*: dictionary with the weight of each error for the custom score calculation.
- **error_format** *(str)*: 'count', 'error_ratio' or 'entity_ratio'. Controls the desired output for TP, FP, BE, LE, etc.  Default value is 'count'.
  - 'count': absolute count of each parameter. 
  - 'error_ratio': precentage with respect to the total errors that each parameter represents.
  - 'entity_ratio': precentage with respect to the total number of ground truth entites that each parameter represents.
- **zero_division** *(str)*: which value to substitute as a metric value when encountering zero division. Should be one of [0,1,"warn"]. "warn" acts as 0, but the warning is raised.
- **suffix** *(boolean)*: True if the IOB tag is a suffix (after type) instead of a prefix (before type), False otherwise. The default value is False, i.e. the IOB tag is a prefix (before type).
- **scheme** *(str)*: the target tagging scheme, which can be one of [IOB1, IOB2, IOE1, IOE2, IOBES, BILOU]. The default value is None.

### Output Values
A dictionary with:
 - Overall error parameter count (or ratio) and resulting scores.
 - A nested dictionary per label with its respective error parameter count (or ratio) and resulting scores

If mode is 'traditional', the error parameters shown are the classical TP, FP and FN. If mode is 'fair' or 'weighted', 
TP remain the same, FP and FN are shown as per the fair definition and additional errors BE, LE and LBE are shown.

### Examples
A comprehensive set of side-by-side examples is shown [here](https://huggingface.co/spaces/hpi-dhc/FairEval/blob/main/HFFE_use_cases.pdf).

Considering the following input annotated sentences:
```python
>>> r1 = ['O', 'O', 'B-PER', 'I-PER', 'O', 'B-PER']
>>> p1 = ['O', 'O', 'B-PER', 'I-PER', 'O', 'O'    ] #1FN
>>> 
>>> r2 = ['O',     'B-INT', 'B-OUT']
>>> p2 = ['B-INT', 'I-INT', 'B-OUT'] #1BE  
>>> 
>>> r3 = ['B-INT', 'I-INT', 'B-OUT']
>>> p3 = ['B-OUT', 'O',     'B-PER'] #1LBE, 1LE   
>>> 
>>> y_true = [r1, r2, r3]
>>> y_pred = [p1, p2, p3]
```

The output for different modes and error_formats is:
```python
>>> faireval.compute(predictions=y_pred, references=y_true, mode='traditional', error_format='count')
{'PER': {'precision': 0.5, 'recall': 0.5, 'f1': 0.5, 'TP': 1, 'FP': 1, 'FN': 1},
 'INT': {'precision': 0.0, 'recall': 0.0, 'f1': 0.0, 'TP': 0, 'FP': 1, 'FN': 2},
 'OUT': {'precision': 0.5, 'recall': 0.5, 'f1': 0.5, 'TP': 1, 'FP': 1, 'FN': 1},
 'overall_precision': 0.4,
 'overall_recall': 0.3333,
 'overall_f1': 0.3636,
 'TP': 2,
 'FP': 3,
 'FN': 4}
```

```python
>>> faireval.compute(predictions=y_pred, references=y_true, mode='traditional', error_format='proportion')
{'PER': {'precision': 0.5, 'recall': 0.5, 'f1': 0.5, 'TP': 1, 'FP': 0.1428, 'FN': 0.1428},
 'INT': {'precision': 0.0, 'recall': 0.0, 'f1': 0.0, 'TP': 0, 'FP': 0.1428, 'FN': 0.2857},
 'OUT': {'precision': 0.5, 'recall': 0.5, 'f1': 0.5, 'TP': 1, 'FP': 0.1428, 'FN': 0.1428},
 'overall_precision': 0.4,
 'overall_recall': 0.3333,
 'overall_f1': 0.3636,
 'TP': 2,
 'FP': 0.4285,
 'FN': 0.5714}
```

```python
>>> faireval.compute(predictions=y_pred, references=y_true, mode='fair', error_format='count')
{'PER': {'precision': 1.0, 'recall': 0.5, 'f1': 0.6666, 'TP': 1, 'FP': 0, 'FN': 1, 'LE': 0, 'BE': 0, 'LBE': 0},
 'INT': {'precision': 0.0, 'recall': 0.0, 'f1': 0.0, 'TP': 0, 'FP': 0, 'FN': 0, 'LE': 0, 'BE': 1, 'LBE': 1},
 'OUT': {'precision': 0.6666, 'recall': 0.6666, 'f1': 0.6666, 'TP': 1, 'FP': 0, 'FN': 0, 'LE': 1, 'BE': 0, 'LBE': 0},
 'overall_precision': 0.5714,
 'overall_recall': 0.4444444444444444,
 'overall_f1': 0.5,
 'TP': 2,
 'FP': 0,
 'FN': 1,
 'LE': 1,
 'BE': 1,
 'LBE': 1}
```

#### Values from Popular Papers
*Examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*

*Under construction*

## Limitations and Bias
*Note any known limitations or biases that the metric has, with links and references if possible.*

*Under construction*

## Citation
Ortmann, Katrin. 2022. Fine-Grained Error Analysis and Fair Evaluation of Labeled Spans. In *Proceedings of the Language Resources and Evaluation Conference (LREC)*, Marseille, France, pages 1400–1407. [PDF](https://aclanthology.org/2022.lrec-1.150.pdf)

```bibtex
@inproceedings{ortmann2022,
    title = {Fine-Grained Error Analysis and Fair Evaluation of Labeled Spans},
    author = {Katrin Ortmann},
    url = {https://aclanthology.org/2022.lrec-1.150},
    year = {2022},
    date = {2022-06-21},
    booktitle = {Proceedings of the Language Resources and Evaluation Conference (LREC)},
    pages = {1400-1407},
    publisher = {European Language Resources Association},
    address = {Marseille, France},
    pubstate = {published},
    type = {inproceedings}
}
```