---
language:
- en
base_model:
- FacebookAI/roberta-large
pipeline_tag: text-classification
---
# Graded Word Sense Disambiguation (WSD) Model

## Model Summary
This model is a **fine-tuned version of RoBERTa-Large** for **Graded Word Sense Disambiguation (WSD)**. It is designed to predict the **degree of applicability** (1-4) of a word sense in context by leveraging **large-scale sense-annotated corpora**. The model is based on the work outlined in:

**Reference Paper:**
Pierluigi Cassotti, Nina Tahmasebi (2025). Sense-specific Historical Word Usage Generation.


This model has been trained to handle **graded WSD tasks**, providing **continuous-valued predictions** instead of hard classification, making it useful for nuanced applications in lexicography, computational linguistics, and historical text analysis.

---

## Model Details
- **Base Model:** `roberta-large`
- **Task:** Graded Word Sense Disambiguation (WSD)
- **Fine-tuning Dataset:** Oxford English Dictionary (OED) sense-annotated corpus
- **Training Steps:**
  - Tokenizer augmented with special tokens (`<t>`, `</t>`) for marking target words in context.
  - Dataset preprocessed with **sense annotations** and **word offsets**.
  - Sentences containing sense-annotated words were split into **train (90%)** and **validation (10%)** sets.
  - **Objective:** Predicting a continuous label representing the applicability of a sense.
  - **Evaluation Metric:** Root Mean Squared Error (RMSE).
- **Batch Size:** 32
- **Learning Rate:** 2e-5
- **Epochs:** 1
- **Optimizer:** AdamW with weight decay of 0.01
- **Evaluation Strategy:** Steps-based (every 10% of the dataset).

---

## Training & Fine-Tuning
Fine-tuning was performed using the **Hugging Face `Trainer` API** with a **custom dataset loader**. The dataset was processed as follows:

1. **Preprocessing**
   - Example sentences were extracted from the OED and augmented with **definitions**.
   - The target word was **highlighted** with special tokens (`<t>`, `</t>`).
   - Each instance was labeled with a **graded similarity score**.

2. **Tokenization & Encoding**
   - Tokenized with `AutoTokenizer.from_pretrained("roberta-large")`.
   - Definitions were concatenated using the `</s></s>` separator for **cross-sentence representation**.

3. **Training Pipeline**
   - Model fine-tuned on the **regression task** with a single **linear output head**.
   - Used **Mean Squared Error (MSE) loss**.
   - Evaluation on validation set using **Root Mean Squared Error (RMSE)**.

---

## Usage
### Example Code
```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("ChangeIsKey/graded-wsd")
model = AutoModelForSequenceClassification.from_pretrained("ChangeIsKey/graded-wsd")

sentence = "The <t>bank</t> of the river was eroding due to the storm."
target_word = "bank"
definition = "The land alongside a river or a stream."

tokenized_input = tokenizer(f"{sentence} </s></s> {definition}", truncation=True, padding=True, return_tensors="pt")
with torch.no_grad():
    output = model(**tokenized_input)
    score = output.logits.item()

print(f"Graded Sense Score: {score}")
```

### Input Format
- Sentence: Contextual usage of the word.
- Target Word: The word to be disambiguated.
- Definition: The dictionary definition of the intended sense.

### Output
- **A continuous score** (between 1 and 4) indicating the **similarity** of the given definition with respect to the word in its current context.

---

## Citation
If you use this model, please cite the following paper:

```
@article{10.1162/tacl_a_00761,
    author = {Cassotti, Pierluigi and Tahmasebi, Nina},
    title = {Sense-specific Historical Word Usage Generation},
    journal = {Transactions of the Association for Computational Linguistics},
    volume = {13},
    pages = {690-708},
    year = {2025},
    month = {07},
    abstract = {Large-scale sense-annotated corpora are important for a range of tasks but are hard to come by. Dictionaries that record and describe the vocabulary of a language often offer a small set of real-world example sentences for each sense of a word. However, on their own, these sentences are too few to be used as diachronic sense-annotated corpora. We propose a targeted strategy for training and evaluating generative models producing historically and semantically accurate word usages given any word, sense definition, and year triple. Our results demonstrate that fine-tuned models can generate usages with the same properties as real-world example sentences from a reference dictionary. Thus the generated usages will be suitable for training and testing computational models where large-scale sense-annotated corpora are needed but currently unavailable.},
    issn = {2307-387X},
    doi = {10.1162/tacl_a_00761},
    url = {https://doi.org/10.1162/tacl\_a\_00761},
    eprint = {https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl\_a\_00761/2535111/tacl\_a\_00761.pdf},
}
```