---
library_name: peft
datasets:
- xnli
license: cc-by-nc-4.0
pipeline_tag: sentence-similarity
---
These are LoRA adaption weights for the [mT5](https://huggingface.co/google/mt5-xxl) encoder.

## Multilingual Sentence T5 (m-ST5)
This model is a multilingual extension of Sentence T5 and was created using the [mT5](https://huggingface.co/google/mt5-xxl) encoder. It is proposed in this [paper](https://arxiv.org/abs/2403.17528).
m-ST5 is an encoder for sentence embedding, and its performance has been verified in cross-lingual semantic textual similarity (STS) and sentence retrieval tasks.

### Training Data
The model was trained on the XNLI dataset.

### Framework versions


- PEFT 0.4.0.dev0

## How to use
0. If you have not installed peft, please do so.
```
pip install -q git+https://github.com/huggingface/transformers.git@main git+https://github.com/huggingface/peft.git
```
1. Load the model.
```
from transformers import MT5EncoderModel
from peft import PeftModel

model =  MT5EncoderModel.from_pretrained("google/mt5-xxl")
model.enable_input_require_grads()
model.gradient_checkpointing_enable()
model: PeftModel = PeftModel.from_pretrained(model, "pkshatech/m-ST5")
```
2. To obtain sentence embedding, use mean pooling.
```
tokenizer = AutoTokenizer.from_pretrained("google/mt5-xxl", use_fast=False)
model.eval()

texts = ["I am a dog.","You are a cat."]
inputs = tokenizer(
    texts,
    padding=True,
    truncation=True,
    return_tensors="pt",
)
outputs = model(**inputs)
last_hidden_state = outputs.last_hidden_state
last_hidden_state[inputs.attention_mask == 0, :] = 0
sent_len = inputs.attention_mask.sum(dim=1, keepdim=True)
sent_emb = last_hidden_state.sum(dim=1) / sent_len
```

## BenchMarks
- Tatoeba: Sentence retrieval tasks with pairs of English sentences and sentences in other languages.
- BUCC: Bitext mining task. It consists of English and one of the 4 languages (German, French, Russian and Chinese).
- XSTS: Cross-lingual semantic textual similarity task.


Please check the paper for details and more.

|       | Tatoeba-14 | Tatoeba-36 | BUCC | XSTS<br>(ar-ar)|XSTS<br>(ar-en)|XSTS<br>(es-es)|XSTS<br>(es-en)|XSTS<br>(tr-en)|
| ----- | :----------: | :----------: | :----: | :---:|:----:|:----:|:----:|:----:|
| m-ST5 | 96.3       | 94.7       | 97.6 | 76.2|78.6|84.4|76.2|75.1|
| LaBSE | 95.3       | 95.0       | 93.5 | 69.1|74.5|80.8|65.5|72.0|