---
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers
- phobert
- french
- sentence-embedding
license: apache-2.0
language:
- fr
- en
metrics:
- pearsonr
- spearmanr
---
## Model Description:
[**french-embedding-LongContext**](https://huggingface.co/dangvantuan/french-embedding-LongContext) is the Embedding Model for French-English language  with context length up to 8096 tokens. This model is a specialized text-embedding trained specifically for the french language, which is built upon [gte-multilingual](Alibaba-NLP/gte-multilingual-base) and trained using the Multi-Negative Ranking Loss, Matryoshka2dLoss and SimilarityLoss.

## Full Model Architecture
```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: BilingualModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)
```
## Training and Fine-tuning process
The model underwent a rigorous four-stage training and fine-tuning process, each tailored to enhance its ability to generate precise and contextually relevant sentence embeddings for the french language. Below is an outline of these stages:
#### Stage 1: Training NLI on dataset XNLI: 
- Dataset: XNLI (fr-en)
- Method: Training using Multi-Negative Ranking Loss and Matryoshka2dLoss. This stage focused on improving the model's ability to discern and rank nuanced differences in sentence semantics.
### Stage 2: Fine-tuning for Semantic Textual Similarity on STS Benchmark
- Dataset: STS-B (fr-en)
- Method: Fine-tuning specifically for the semantic textual similarity benchmark using Siamese BERT-Networks configured with the 'sentence-transformers' library. This stage honed the model's precision in capturing semantic similarity across various types of french texts.


## Usage:

Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:

```
pip install -U sentence-transformers
```

Then you can use the model like this:

```python
from sentence_transformers import SentenceTransformer
sentences = ["Paris est une capitale de la France", "Paris is a capital of France"]



model = SentenceTransformer('dangvantuan/french-embedding-LongContext', trust_remote_code=True)
embeddings = model.encode(sentences)
print(embeddings)

```


## Evaluation






## Citation


	@article{reimers2019sentence,
	   title={Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
	   author={Nils Reimers, Iryna Gurevych},
	   journal={https://arxiv.org/abs/1908.10084},
	   year={2019}
	}


    @article{zhang2024mgte,
      title={mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval},
      author={Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Wen and Dai, Ziqi and Tang, Jialong and Lin, Huan and Yang, Baosong and Xie, Pengjun and Huang, Fei and others},
      journal={arXiv preprint arXiv:2407.19669},
      year={2024}
    }
    
    @article{li2023towards,
      title={Towards general text embeddings with multi-stage contrastive learning},
      author={Li, Zehan and Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Pengjun and Zhang, Meishan},
      journal={arXiv preprint arXiv:2308.03281},
      year={2023}
    }
    
    @article{li20242d,
      title={2d matryoshka sentence embeddings},
      author={Li, Xianming and Li, Zongxi and Li, Jing and Xie, Haoran and Li, Qing},
      journal={arXiv preprint arXiv:2402.14776},
      year={2024}
    }