File size: 4,570 Bytes
c7656cd 6196e7b c7656cd 6196e7b c7656cd 6196e7b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 |
---
tags:
- text2text-generation
- definition-modeling
metrics:
- rouge, bleu, bert-f1
model-index:
- name: flan-t5-definition-en-base
results: []
language:
- en
widget:
- text: "He ate a sweet apple. What is the definition of apple?"
example_title: "Definition generation"
- text: "The paper contains a number of original ideas about color perception. What is the definition of original?"
example_title: "Definition generation"
license: cc-by-sa-4.0
datasets:
- marksverdhei/wordnet-definitions-en-2021
---
# FLAN-T5-Definition Base
This model is a version of [FLAN-T5 Base](https://huggingface.co/google/flan-t5-base) finetuned on a dataset of English definitions and usage examples.
It generates definitions of English words in context.
Its input is the usage example and the instruction question "What is the definiton of TARGET_WORD?"
## Model description
See details in the paper `Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis` (ACL'2023) by Mario Giulianelli, Iris Luden, Raquel Fernandez and Andrey Kutuzov.
## Intended uses & limitations
The model is intended for research purposes, as a source of contextualized dictionary-like lexical definitions.
The fine-tuning datasets were limited to English.
Although the original FLAN-T5 is a multilingual model, we did not thoroughly evaluate its ability to generate definitions in languages other than English.
Generated definitions can contain all sorts of biases and stereotypes, stemming from the underlying language model.
## Training and evaluation data
Three datasets were used to fine-tune the model:
- *WordNet* ([Ishiwatari et al., NAACL 2019](https://aclanthology.org/N19-1350/)), also [available on HF](https://huggingface.co/datasets/marksverdhei/wordnet-definitions-en-2021)
- *Oxford dictionary or CHA* ([Gadetsky et al., ACL 2018](https://aclanthology.org/P18-2043/))
- English subset of *CodWoE* ([Mickus et al., SemEval 2022](https://aclanthology.org/2022.semeval-1.1/))
FLAN-T5-Definition Base achieves the following results on the WordNet test set:
- ROUGE-L: 23.19
- BLEU: 8.80
- BERT-F1: 87.49
FLAN-T5-Definition Base achieves the following results on the Oxford dictionary test set:
- ROUGE-L: 17.25
- BLEU: 3.71
- BERT-F1: 86.44
## Training procedure
FLAN-T5 Base was fine-tuned in a sequence-to-sequence mode on examples of contextualized dictionary definitions.
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 15.0
### Training results
| Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Gen Len |
|:-------------:|:-----:|:-----:|:---------------:|:-------:|:------:|:-------:|:---------:|:-------:|
| 2.5645 | 1.0 | 2740 | 2.2535 | 24.4437 | 6.4189 | 22.7949 | 22.7909 | 11.4969 |
| 2.3501 | 2.0 | 5480 | 2.1642 | 25.6642 | 7.289 | 23.8689 | 23.8749 | 11.7150 |
| 2.2516 | 3.0 | 8220 | 2.1116 | 26.4562 | 7.8955 | 24.6275 | 24.6376 | 11.7441 |
| 2.1806 | 4.0 | 10960 | 2.0737 | 27.0392 | 8.2393 | 25.1555 | 25.1641 | 11.7930 |
| 2.1233 | 5.0 | 13700 | 2.0460 | 27.2709 | 8.4244 | 25.3847 | 25.4003 | 11.9014 |
| 2.0765 | 6.0 | 16440 | 2.0236 | 27.5456 | 8.6096 | 25.6321 | 25.6462 | 11.8113 |
| 2.0371 | 7.0 | 19180 | 2.0047 | 27.7209 | 8.7277 | 25.7871 | 25.8084 | 11.6875 |
| 2.0036 | 8.0 | 21920 | 1.9918 | 28.0431 | 8.9863 | 26.1072 | 26.1198 | 11.5487 |
| 1.9752 | 9.0 | 24660 | 1.9788 | 28.1807 | 9.0219 | 26.1692 | 26.1886 | 11.7939 |
| 1.9513 | 10.0 | 27400 | 1.9702 | 28.3204 | 9.1572 | 26.2955 | 26.3029 | 11.5936 |
| 1.9309 | 11.0 | 30140 | 1.9640 | 28.4289 | 9.2845 | 26.4006 | 26.418 | 11.8371 |
| 1.9144 | 12.0 | 32880 | 1.9571 | 28.4504 | 9.3406 | 26.4273 | 26.4384 | 11.6201 |
| 1.9013 | 13.0 | 35620 | 1.9544 | 28.6319 | 9.3682 | 26.605 | 26.613 | 11.7067 |
| 1.8914 | 14.0 | 38360 | 1.9512 | 28.6435 | 9.3976 | 26.5839 | 26.5918 | 11.7307 |
| 1.8866 | 15.0 | 41100 | 1.9509 | 28.6111 | 9.3857 | 26.551 | 26.5648 | 11.7470 |
### Framework versions
- Transformers 4.24.0
- Pytorch 1.11.0
- Datasets 2.3.2
- Tokenizers 0.12.1
|