Llama3Dictionary-merge

FrancescoPeriti/Llama3Dictionary-merge integrates the fine-tuned FrancescoPeriti/Llama3Dictionary with the original meta-llama/Meta-Llama-3-8B-Instruct.

Model Description

This model is fine-tuned on English datasets of sense definitions. Given a target word and a usage example, the model generates a sense definition for the target word in-context.

You can find more details in the paper Automatically Generated Definitions and their utility for Modeling Word Meaning by Francesco Periti, David Alfter, Nina Tahmasebi. The repository of our project is https://github.com/FrancescoPeriti/LlamaDictionary.

Uses

The model is designed for research purposes and is conceived to work like a dictionary. However, given a word and an example usage, users don't choose from a list of definitions (as in a traditional dictionary); instead, the model directly provides the sense definition for the word in-context.

Bias, Risks, and Limitations

The fine-tuning datasets were limited to English, and generated definitions may reflect biases and stereotypes inherent in the underlying language model.

How to Get Started with the Model

from transformers import pipeline
from transformers import AutoTokenizer

model_name = "meta-llama/Meta-Llama-3-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name,
                                          padding_side="left",
                                          add_eos_token=True,
                                          add_bos_token=True)
tokenizer.pad_token = tokenizer.eos_token

# end of sequence for stop condition
eos_tokens = [tokenizer.eos_token_id]

pipe = pipeline("text-generation", model="FrancescoPeriti/Llama3Dictionary-merge", tokenizer=tokenizer, device_map="auto")

chat = [
    {"role": "system", 
     "content": "You are a lexicographer familiar with providing concise definitions of word meanings."}, 
    {"role": "user", 
     "content": 'Please provide a concise definition for the meaning of the word "jam" in the following sentence: The traffic jam on the highway made everyone late for work.'}
]

prompt = pipe.tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)

pipe.tokenizer.padding_side='left'
pipe.tokenizer.add_special_tokens = True
pipe.tokenizer.add_eos_token = True
pipe.tokenizer.add_bos_token = True

eos_tokens = [26, 2652, 13, 662, 128009] # [';', ' ;', '.', ' .']
outputs = pipe(prompt, max_length = 512, forced_eos_token_id = eos_tokens,
               max_time = 4.5, eos_token_id = eos_tokens, temperature = 0.00001,
               pad_token_id = pipe.tokenizer.eos_token_id)

print(outputs[0]["generated_text"])

Citation

Francesco Periti, David Alfter, and Nina Tahmasebi. 2024. Automatically Generated Definitions and their utility for Modeling Word Meaning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14008–14026, Miami, Florida, USA. Association for Computational Linguistics.

BibTeX:

@inproceedings{periti2024automatically,
    title = {{Automatically Generated Definitions and their utility for Modeling Word Meaning}},
    author = "Periti, Francesco  and Alfter, David  and Tahmasebi, Nina",
    editor = "Al-Onaizan, Yaser  and Bansal, Mohit  and Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-main.776",
    pages = "14008--14026",
    abstract = "Modeling lexical semantics is a challenging task, often suffering from interpretability pitfalls. In this paper, we delve into the generation of dictionary-like sense definitions and explore their utility for modeling word meaning. We fine-tuned two Llama models and include an existing T5-based model in our evaluation. Firstly, we evaluate the quality of the generated definitions on existing English benchmarks, setting new state-of-the-art results for the Definition Generation task. Next, we explore the use of definitions generated by our models as intermediate representations subsequently encoded as sentence embeddings. We evaluate this approach on lexical semantics tasks such as the Word-in-Context, Word Sense Induction, and Lexical Semantic Change, setting new state-of-the-art results in all three tasks when compared to unsupervised baselines.",
}