README.md

---
license: cc-by-sa-4.0
language:
- en
library_name: transformers
pipeline_tag: text2text-generation
tags:
- text-generation-inference
base_model:
- meta-llama/Meta-Llama-3-8B-Instruct
new_version: FrancescoPeriti/Llama3Dictionary-merge
---

# Llama3Dictionary

<!-- Provide a quick summary of what the model is/does. -->
```FrancescoPeriti/Llama3Dictionary``` is a fine-tuned version of the ```meta-llama/Meta-Llama-3-8B-Instruct```. 
Thus, to use it, visit the AI at Meta website, accept the Meta License, and submit the [form](https://llama.meta.com/llama-downloads/).

You will need to login with your hugginface token (```[HF-TOKEN]```, in the following).


### Model Description
This model is fine-tuned on English datasets of sense definitions. Given a target word and a usage example, the model generates a sense definition for the target word in-context.

You can find more details in the paper [Automatically Generated Definitions and their utility for Modeling Word Meaning](https://aclanthology.org/2024.emnlp-main.776/) by Francesco Periti, David Alfter, Nina Tahmasebi.
The repository of our project is [https://github.com/FrancescoPeriti/LlamaDictionary](https://github.com/FrancescoPeriti/LlamaDictionary).

## Uses
The model is designed for research purposes and is conceived to work like a dictionary. 
However, given a word and an example usage, users don't choose from a list of definitions (as in a traditional dictionary); instead, the model directly provides the sense definition for the word in-context.

<!-- ### Direct Use -->
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

<!-- ### Downstream Use [optional]-->
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

## Bias, Risks, and Limitations
The fine-tuning datasets were limited to English, and generated definitions may reflect biases and stereotypes inherent in the underlying language model.

## How to Get Started with the Model
```python
import torch
import warnings
from peft import PeftModel # parameter-efficient fine-tuning
from datasets import Dataset
from huggingface_hub import login
from typing import (Literal, Sequence,TypedDict)
from transformers import AutoTokenizer, AutoModelForCausalLM

login([HF-TOKEN]) # e.g., hf_aGPI...ELal

model_name = "meta-llama/Meta-Llama-3-8B-Instruct" # chat model
ft_model_name = "FrancescoPeriti/Llama3Dictionary" # fine-tuned model

# load models
chat_model = AutoModelForCausalLM.from_pretrained(model_name, device_map='auto')
lama3dictionary = PeftModel.from_pretrained(chat_model, ft_model_name)
lama3dictionary.eval()

# load tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    padding_side="left",
    add_eos_token=True,
    add_bos_token=True,
)
tokenizer.pad_token = tokenizer.eos_token

# end of sequence for stop condition
eos_tokens = [tokenizer.encode(token, add_special_tokens=False)[0] 
              for token in [';', ' ;', '.', ' .']]
eos_tokens.append(tokenizer.eos_token_id)

# chat format
Role = Literal["system", "user"]

class Message(TypedDict):
    role: Role
    content: str

Dialog = Sequence[Message]

# load dataset
examples = [{'target': 'jam', 'example': 'The traffic jam on the highway made everyone late for work.'},
            {'target': 'jam', 'example': 'I spread a generous layer of strawberry jam on my toast this morning'}]
dataset = Dataset.from_list(examples)

# apply template
def apply_chat_template(tokenizer, dataset):
    system_message = "You are a lexicographer familiar with providing concise definitions of word meanings."
    template = 'Please provide a concise definition for the meaning of the word "{}" in the following sentence: {}'
                
    def apply_chat_template_func(record):
        dialog: Dialog = (Message(role='system', content=system_message),
                          Message(role='user', content=template.format(record['target'], record['example'])))
        prompt = tokenizer.decode(tokenizer.apply_chat_template(dialog, add_generation_prompt=True))
        return {'text': prompt}
    
    return dataset.map(apply_chat_template_func)

dataset = apply_chat_template(tokenizer, dataset)

# tokenization
max_length = 512

def formatting_func(record):
    return record['text']

def tokenization(dataset):
    result = tokenizer(formatting_func(dataset),
                       truncation=True,
                       max_length=max_length,
                       padding="max_length",
                       add_special_tokens=False)
    return result

tokenized_dataset = dataset.map(tokenization)

# definition generation
batch_size = 32
max_time = 4.5 # sec

sense_definitions = list()
with torch.no_grad():
    for i in range(0, len(tokenized_dataset), batch_size):
        batch = tokenized_dataset[i:i + batch_size]

        model_input = dict()
        for k in ['input_ids', 'attention_mask']:
            model_input[k] = torch.tensor(batch[k]).to('cuda')

        output_ids = lama3dictionary.generate(**model_input,
                                       max_length = max_length,
                                       forced_eos_token_id = eos_tokens,
                                       max_time = max_time * batch_size,
                                       eos_token_id = eos_tokens,
                                       temperature = 0.00001,
                                       pad_token_id = tokenizer.eos_token_id)

        answers = tokenizer.batch_decode(output_ids, skip_special_tokens=True)

        for j, answer in enumerate(answers):
            answer = answer.split('\n')[-1].strip(" .,;:")
            if len(answer) == 0:
                warnings.warn("Something went wrong. The input example might be too long; try reducing it.")
            sense_definitions.append(answer.replace('\n', ' ') + '\n')

# output
dataset = dataset.add_column('definition', sense_definitions)
for row in dataset:
    print(f"Target: {row['target']}\nExample: {row['example']}\nSense definition: {row['definition']}")
```

## Citation

Francesco Periti, David Alfter, and Nina Tahmasebi. 2024. [Automatically Generated Definitions and their utility for Modeling Word Meaning](https://aclanthology.org/2024.emnlp-main.776/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14008–14026, Miami, Florida, USA. Association for Computational Linguistics.

**BibTeX:**
```
@inproceedings{periti2024automatically,
    title = {{Automatically Generated Definitions and their utility for Modeling Word Meaning}},
    author = "Periti, Francesco  and Alfter, David  and Tahmasebi, Nina",
    editor = "Al-Onaizan, Yaser  and Bansal, Mohit  and Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-main.776",
    pages = "14008--14026",
    abstract = "Modeling lexical semantics is a challenging task, often suffering from interpretability pitfalls. In this paper, we delve into the generation of dictionary-like sense definitions and explore their utility for modeling word meaning. We fine-tuned two Llama models and include an existing T5-based model in our evaluation. Firstly, we evaluate the quality of the generated definitions on existing English benchmarks, setting new state-of-the-art results for the Definition Generation task. Next, we explore the use of definitions generated by our models as intermediate representations subsequently encoded as sentence embeddings. We evaluate this approach on lexical semantics tasks such as the Word-in-Context, Word Sense Induction, and Lexical Semantic Change, setting new state-of-the-art results in all three tasks when compared to unsupervised baselines.",
}
```