|
--- |
|
tags: |
|
- text2text-generation |
|
- definition-modeling |
|
metrics: |
|
- rouge, bleu, bert-f1 |
|
model-index: |
|
- name: flan-t5-definition-en-base |
|
results: [] |
|
language: |
|
- en |
|
widget: |
|
- text: "He ate a sweet apple. What is the definition of apple?" |
|
example_title: "Definition generation" |
|
- text: "The paper contains a number of original ideas about color perception. What is the definition of original?" |
|
example_title: "Definition generation" |
|
license: cc-by-sa-4.0 |
|
datasets: |
|
- marksverdhei/wordnet-definitions-en-2021 |
|
--- |
|
|
|
# FLAN-T5-Definition Base |
|
|
|
This model is a version of [FLAN-T5 Base](https://huggingface.co/google/flan-t5-base) finetuned on a dataset of English definitions and usage examples. |
|
|
|
It generates definitions of English words in context. |
|
Its input is the usage example and the instruction question "What is the definiton of TARGET_WORD?" |
|
|
|
This project is a collaboration between the [Dialogue Modelling Group](https://dmg-illc.github.io/dmg/) at the University of Amsterdam |
|
and the [Language Technology Group](https://www.mn.uio.no/ifi/english/research/groups/ltg/) at the University of Oslo. |
|
|
|
## Sizes: |
|
- [FLAN-T5-Definition Base (250M parameters)](https://huggingface.co/ltg/flan-t5-definition-en-base) |
|
- [FLAN-T5-Definition XL (3B parameters)](https://huggingface.co/ltg/flan-t5-definition-en-xl) |
|
|
|
## Model description |
|
|
|
See details in the paper `Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis` (ACL'2023) by Mario Giulianelli, Iris Luden, Raquel Fernandez and Andrey Kutuzov. |
|
|
|
## Intended uses & limitations |
|
|
|
The model is intended for research purposes, as a source of contextualized dictionary-like lexical definitions. |
|
|
|
The fine-tuning datasets were limited to English. |
|
Although the original FLAN-T5 is a multilingual model, we did not thoroughly evaluate its ability to generate definitions in languages other than English. |
|
|
|
Generated definitions can contain all sorts of biases and stereotypes, stemming from the underlying language model. |
|
|
|
## Training and evaluation data |
|
|
|
Three datasets were used to fine-tune the model: |
|
- *WordNet* ([Ishiwatari et al., NAACL 2019](https://aclanthology.org/N19-1350/)), also [available on HF](https://huggingface.co/datasets/marksverdhei/wordnet-definitions-en-2021) |
|
- *Oxford dictionary or CHA* ([Gadetsky et al., ACL 2018](https://aclanthology.org/P18-2043/)) |
|
- English subset of *CodWoE* ([Mickus et al., SemEval 2022](https://aclanthology.org/2022.semeval-1.1/)) |
|
|
|
FLAN-T5-Definition Base achieves the following results on the WordNet test set: |
|
- BLEU: 10.38 |
|
- ROUGE-L: 27.17 |
|
- BERT-F1: 88.22 |
|
|
|
FLAN-T5-Definition Base achieves the following results on the Oxford dictionary test set: |
|
- BLEU: 7.18 |
|
- ROUGE-L: 23.04 |
|
- BERT-F1: 86.90 |
|
|
|
|
|
## Training procedure |
|
FLAN-T5 Base was fine-tuned in a sequence-to-sequence mode on examples of contextualized dictionary definitions. |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 5e-05 |
|
- train_batch_size: 64 |
|
- eval_batch_size: 64 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- num_epochs: 15.0 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Gen Len | |
|
|:-------------:|:-----:|:-----:|:---------------:|:-------:|:------:|:-------:|:---------:|:-------:| |
|
| 2.5645 | 1.0 | 2740 | 2.2535 | 24.4437 | 6.4189 | 22.7949 | 22.7909 | 11.4969 | |
|
| 2.3501 | 2.0 | 5480 | 2.1642 | 25.6642 | 7.289 | 23.8689 | 23.8749 | 11.7150 | |
|
| 2.2516 | 3.0 | 8220 | 2.1116 | 26.4562 | 7.8955 | 24.6275 | 24.6376 | 11.7441 | |
|
| 2.1806 | 4.0 | 10960 | 2.0737 | 27.0392 | 8.2393 | 25.1555 | 25.1641 | 11.7930 | |
|
| 2.1233 | 5.0 | 13700 | 2.0460 | 27.2709 | 8.4244 | 25.3847 | 25.4003 | 11.9014 | |
|
| 2.0765 | 6.0 | 16440 | 2.0236 | 27.5456 | 8.6096 | 25.6321 | 25.6462 | 11.8113 | |
|
| 2.0371 | 7.0 | 19180 | 2.0047 | 27.7209 | 8.7277 | 25.7871 | 25.8084 | 11.6875 | |
|
| 2.0036 | 8.0 | 21920 | 1.9918 | 28.0431 | 8.9863 | 26.1072 | 26.1198 | 11.5487 | |
|
| 1.9752 | 9.0 | 24660 | 1.9788 | 28.1807 | 9.0219 | 26.1692 | 26.1886 | 11.7939 | |
|
| 1.9513 | 10.0 | 27400 | 1.9702 | 28.3204 | 9.1572 | 26.2955 | 26.3029 | 11.5936 | |
|
| 1.9309 | 11.0 | 30140 | 1.9640 | 28.4289 | 9.2845 | 26.4006 | 26.418 | 11.8371 | |
|
| 1.9144 | 12.0 | 32880 | 1.9571 | 28.4504 | 9.3406 | 26.4273 | 26.4384 | 11.6201 | |
|
| 1.9013 | 13.0 | 35620 | 1.9544 | 28.6319 | 9.3682 | 26.605 | 26.613 | 11.7067 | |
|
| 1.8914 | 14.0 | 38360 | 1.9512 | 28.6435 | 9.3976 | 26.5839 | 26.5918 | 11.7307 | |
|
| 1.8866 | 15.0 | 41100 | 1.9509 | 28.6111 | 9.3857 | 26.551 | 26.5648 | 11.7470 | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.24.0 |
|
- Pytorch 1.11.0 |
|
- Datasets 2.3.2 |
|
- Tokenizers 0.12.1 |
|
|
