|
--- |
|
language: en |
|
--- |
|
# UnifiedQA-Reddit-SYAC |
|
|
|
This is an abstractive title answering (TA) / clickbait spoiling model. |
|
This is a variant of [allenai/unifiedqa-t5-large](https://huggingface.co/allenai/unifiedqa-t5-large), fine-tuned on the Reddit SYAC dataset. |
|
The model was trained as part of my masters thesis: |
|
|
|
_Abstractive title answering for clickbait content_ |
|
|
|
|
|
### Disinformation |
|
This model has the proven capability of generating, and hallucinating false information. |
|
Any use of a TA system such as this one should be with knowledge of this risk. |
|
|
|
|
|
## Performance |
|
|
|
### Intrinsic |
|
|
|
The following scores is the result of intrinsic evaluation on the Reddit SYAC test set. |
|
We used a max input length of 2048 and truncated the tokens exceeding this limit. |
|
|
|
| rouge1 | rouge2 | rougeL | bleu | meteor | |
|
|:----------|:----------|:----------|:----------|:---------| |
|
| **44.58** | **23.89** | **43.45** | 17.46 | 36.22 | |
|
|
|
|
|
### Qualtiy |
|
Using human evaluation, we measured model performance by asking the evaluators to rate the models |
|
on a scale from 1 to 5 on how good their generated answer was for a given clickbait article. |
|
|
|
Mean quality = 4.065 |
|
|
|
### Factuality |
|
We included a factuality assessment to address the issue of generating false information. |
|
Human raters were asked to place each output in the categories "True", "Irrelevant", and "False". |
|
|
|
| True | Irrelevant | False | |
|
|:-------:|:----------:|:--------:| |
|
| 85% | 7.5% | 7.5% | |
|
|
|
## Cite |
|
|
|
If you use this model, please cite my master's thesis |
|
|
|
``` |
|
@mastersthesis{heiervang2022AbstractiveTA |
|
title={Abstractive title answering for clickbait content}, |
|
author={Markus Sverdvik Heiervang}, |
|
publisher={University of Oslo, Department of Informatics}, |
|
year={2022} |
|
} |
|
``` |
