Model Card for LuxEmbedder

Model Summary

LuxEmbedder is a sentence-transformers model that transforms sentences and paragraphs into 768-dimensional dense vectors, enabling tasks like clustering and semantic search, with a primary focus on Luxembourgish. Leveraging a cross-lingual approach, LuxEmbedder effectively handles Luxembourgish text while also mapping input from 108 other languages into a shared embedding space. For the full list of supported languages, refer to the sentence-transformers/LaBSE documentation, as LaBSE served as the foundation for LuxEmbedder.

This model was introduced in LuxEmbedder: A Cross-Lingual Approach to Enhanced Luxembourgish Sentence Embeddings (Philippy et al., 2024). It addresses the challenges of limited parallel data for Luxembourgish by creating LuxAlign, a high-quality, human-generated parallel dataset, which forms the basis for LuxEmbedder’s competitive performance across cross-lingual and monolingual tasks for Luxembourgish.

With the release of LuxEmbedder, we also provide a Luxembourgish paraphrase detection benchmark, ParaLux to encourage further exploration and development in NLP for Luxembourgish.

• Model type: Sentence Embedding Model
• Language(s) (NLP): Luxembourgish + 108 additional languages
• License: Creative Commons Attribution Non Commercial 4.0 International (CC BY-NC 4.0)
• Architecture: Based on LaBSE
• Paper: LuxEmbedder: A Cross-Lingual Approach to Enhanced Luxembourgish Sentence Embeddings (Philippy et al., 2024)
• Repository: https://github.com/fredxlpy/LuxEmbedder

Example Usage

pip install -U sentence-transformers

from sentence_transformers import SentenceTransformer, util
import numpy as np
import pandas as pd

# Load the model
model = SentenceTransformer('fredxlpy/LuxEmbedder')

# Example sentences
data = pd.DataFrame({
    "id": ["lb1", "lb2", "lb3", "en1", "en2", "en3", "zh1", "zh2", "zh3"],
    "text": [
        "Moien, wéi geet et?",         # Luxembourgish: Hello, how are you?
        "D'Wieder ass haut schéin.",   # Luxembourgish: The weather is beautiful today.
        "Ech schaffen am Büro.",       # Luxembourgish: I work in the office.
        "Hello, how are you?",         
        "The weather is great today.", 
        "I work in an office.",        
        "你好, 你怎么样?",               # Chinese: Hello, how are you?
        "今天天气很好.",                 # Chinese: The weather is very good today.
        "我在办公室工作."                # Chinese: I work in an office.
    ]
})

# Encode the sentences to obtain sentence embeddings
embeddings = model.encode(data["text"].tolist(), convert_to_tensor=True)

# Compute the cosine similarity matrix
cosine_similarity_matrix = util.cos_sim(embeddings, embeddings).cpu().numpy()

# Create a DataFrame for the similarity matrix with "id" as row and column labels
similarity_df = pd.DataFrame(
    np.round(cosine_similarity_matrix, 2),
    index=data["id"],
    columns=data["id"]
)

# Display the similarity matrix
print("Cosine Similarity Matrix:")
print(similarity_df)

# Cosine Similarity Matrix:
# id    lb1   lb2   lb3   en1   en2   en3   zh1   zh2   zh3
# id                                                       
# lb1  1.00  0.60  0.42  0.96  0.59  0.40  0.95  0.62  0.43
# lb2  0.60  1.00  0.41  0.56  0.99  0.39  0.56  0.99  0.42
# lb3  0.42  0.41  1.00  0.44  0.42  0.99  0.46  0.43  0.99
# en1  0.96  0.56  0.44  1.00  0.55  0.43  0.99  0.58  0.46
# en2  0.59  0.99  0.42  0.55  1.00  0.40  0.55  0.99  0.43
# en3  0.40  0.39  0.99  0.43  0.40  1.00  0.44  0.41  0.99
# zh1  0.95  0.56  0.46  0.99  0.55  0.44  1.00  0.58  0.47
# zh2  0.62  0.99  0.43  0.58  0.99  0.41  0.58  1.00  0.44
# zh3  0.43  0.42  0.99  0.46  0.43  0.99  0.47  0.44  1.00

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 768, 'bias': True, 'activation_function': 'torch.nn.modules.activation.Tanh'})
  (3): Normalize()
)

Citation

@misc{philippy2024luxembedder,
      title={LuxEmbedder: A Cross-Lingual Approach to Enhanced Luxembourgish Sentence Embeddings}, 
      author={Fred Philippy and Siwen Guo and Jacques Klein and Tegawendé F. Bissyandé},
      year={2024},
      eprint={2412.03331},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.03331}, 
}