luismsgomes commited on
Commit
5534b22
verified
1 Parent(s): 2785131

added bibtex

Browse files
Files changed (1) hide show
  1. README.md +145 -132
README.md CHANGED
@@ -1,132 +1,145 @@
1
- ---
2
- language: pt
3
- license: mit
4
- library_name: sentence-transformers
5
- pipeline_tag: sentence-similarity
6
- tags:
7
- - sentence-transformers
8
- - feature-extraction
9
- - sentence-similarity
10
- - transformers
11
-
12
- ---
13
-
14
- # Serafim 900m Portuguese (PT) Sentence Encoder
15
-
16
- This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 1536 dimensional dense vector space and can be used for tasks like clustering or semantic search.
17
-
18
- <!--- Describe your model here -->
19
-
20
- ## Usage (Sentence-Transformers)
21
-
22
- Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
23
-
24
- ```
25
- pip install -U sentence-transformers
26
- ```
27
-
28
- Then you can use the model like this:
29
-
30
- ```python
31
- from sentence_transformers import SentenceTransformer
32
- sentences = ["This is an example sentence", "Each sentence is converted"]
33
-
34
- model = SentenceTransformer('PORTULAN/serafim-900m-portuguese-pt-sentence-encoder')
35
- embeddings = model.encode(sentences)
36
- print(embeddings)
37
- ```
38
-
39
-
40
-
41
- ## Usage (HuggingFace Transformers)
42
- Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
43
-
44
- ```python
45
- from transformers import AutoTokenizer, AutoModel
46
- import torch
47
-
48
-
49
- #Mean Pooling - Take attention mask into account for correct averaging
50
- def mean_pooling(model_output, attention_mask):
51
- token_embeddings = model_output[0] #First element of model_output contains all token embeddings
52
- input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
53
- return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
54
-
55
-
56
- # Sentences we want sentence embeddings for
57
- sentences = ['This is an example sentence', 'Each sentence is converted']
58
-
59
- # Load model from HuggingFace Hub
60
- tokenizer = AutoTokenizer.from_pretrained('PORTULAN/serafim-900m-portuguese-pt-sentence-encoder')
61
- model = AutoModel.from_pretrained('PORTULAN/serafim-900m-portuguese-pt-sentence-encoder')
62
-
63
- # Tokenize sentences
64
- encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
65
-
66
- # Compute token embeddings
67
- with torch.no_grad():
68
- model_output = model(**encoded_input)
69
-
70
- # Perform pooling. In this case, mean pooling.
71
- sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
72
-
73
- print("Sentence embeddings:")
74
- print(sentence_embeddings)
75
- ```
76
-
77
-
78
-
79
- ## Evaluation Results
80
-
81
- <!--- Describe how your model was evaluated -->
82
-
83
- For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=PORTULAN/serafim-900m-portuguese-pt-sentence-encoder)
84
-
85
-
86
- ## Training
87
- The model was trained with the parameters:
88
-
89
- **DataLoader**:
90
-
91
- `torch.utils.data.dataloader.DataLoader` of length 1183 with parameters:
92
- ```
93
- {'batch_size': 16, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
94
- ```
95
-
96
- **Loss**:
97
-
98
- `sentence_transformers.losses.CoSENTLoss.CoSENTLoss` with parameters:
99
- ```
100
- {'scale': 20.0, 'similarity_fct': 'pairwise_cos_sim'}
101
- ```
102
-
103
- Parameters of the fit()-Method:
104
- ```
105
- {
106
- "epochs": 10,
107
- "evaluation_steps": 119,
108
- "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
109
- "max_grad_norm": 1,
110
- "optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
111
- "optimizer_params": {
112
- "lr": 1e-06
113
- },
114
- "scheduler": "WarmupLinear",
115
- "steps_per_epoch": 1183,
116
- "warmup_steps": 1183,
117
- "weight_decay": 0.01
118
- }
119
- ```
120
-
121
-
122
- ## Full Model Architecture
123
- ```
124
- SentenceTransformer(
125
- (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: DebertaV2Model
126
- (1): Pooling({'word_embedding_dimension': 1536, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
127
- )
128
- ```
129
-
130
- ## Citing & Authors
131
-
132
- <!--- Describe where people can find more information -->
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: pt
3
+ license: mit
4
+ library_name: sentence-transformers
5
+ pipeline_tag: sentence-similarity
6
+ tags:
7
+ - sentence-transformers
8
+ - feature-extraction
9
+ - sentence-similarity
10
+ - transformers
11
+
12
+ ---
13
+
14
+ # Serafim 900m Portuguese (PT) Sentence Encoder
15
+
16
+ This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 1536 dimensional dense vector space and can be used for tasks like clustering or semantic search.
17
+
18
+ <!--- Describe your model here -->
19
+
20
+ ## Usage (Sentence-Transformers)
21
+
22
+ Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
23
+
24
+ ```
25
+ pip install -U sentence-transformers
26
+ ```
27
+
28
+ Then you can use the model like this:
29
+
30
+ ```python
31
+ from sentence_transformers import SentenceTransformer
32
+ sentences = ["This is an example sentence", "Each sentence is converted"]
33
+
34
+ model = SentenceTransformer('PORTULAN/serafim-900m-portuguese-pt-sentence-encoder')
35
+ embeddings = model.encode(sentences)
36
+ print(embeddings)
37
+ ```
38
+
39
+
40
+
41
+ ## Usage (HuggingFace Transformers)
42
+ Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
43
+
44
+ ```python
45
+ from transformers import AutoTokenizer, AutoModel
46
+ import torch
47
+
48
+
49
+ #Mean Pooling - Take attention mask into account for correct averaging
50
+ def mean_pooling(model_output, attention_mask):
51
+ token_embeddings = model_output[0] #First element of model_output contains all token embeddings
52
+ input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
53
+ return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
54
+
55
+
56
+ # Sentences we want sentence embeddings for
57
+ sentences = ['This is an example sentence', 'Each sentence is converted']
58
+
59
+ # Load model from HuggingFace Hub
60
+ tokenizer = AutoTokenizer.from_pretrained('PORTULAN/serafim-900m-portuguese-pt-sentence-encoder')
61
+ model = AutoModel.from_pretrained('PORTULAN/serafim-900m-portuguese-pt-sentence-encoder')
62
+
63
+ # Tokenize sentences
64
+ encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
65
+
66
+ # Compute token embeddings
67
+ with torch.no_grad():
68
+ model_output = model(**encoded_input)
69
+
70
+ # Perform pooling. In this case, mean pooling.
71
+ sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
72
+
73
+ print("Sentence embeddings:")
74
+ print(sentence_embeddings)
75
+ ```
76
+
77
+
78
+
79
+ ## Evaluation Results
80
+
81
+ <!--- Describe how your model was evaluated -->
82
+
83
+ For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=PORTULAN/serafim-900m-portuguese-pt-sentence-encoder)
84
+
85
+
86
+ ## Training
87
+ The model was trained with the parameters:
88
+
89
+ **DataLoader**:
90
+
91
+ `torch.utils.data.dataloader.DataLoader` of length 1183 with parameters:
92
+ ```
93
+ {'batch_size': 16, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
94
+ ```
95
+
96
+ **Loss**:
97
+
98
+ `sentence_transformers.losses.CoSENTLoss.CoSENTLoss` with parameters:
99
+ ```
100
+ {'scale': 20.0, 'similarity_fct': 'pairwise_cos_sim'}
101
+ ```
102
+
103
+ Parameters of the fit()-Method:
104
+ ```
105
+ {
106
+ "epochs": 10,
107
+ "evaluation_steps": 119,
108
+ "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
109
+ "max_grad_norm": 1,
110
+ "optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
111
+ "optimizer_params": {
112
+ "lr": 1e-06
113
+ },
114
+ "scheduler": "WarmupLinear",
115
+ "steps_per_epoch": 1183,
116
+ "warmup_steps": 1183,
117
+ "weight_decay": 0.01
118
+ }
119
+ ```
120
+
121
+
122
+ ## Full Model Architecture
123
+ ```
124
+ SentenceTransformer(
125
+ (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: DebertaV2Model
126
+ (1): Pooling({'word_embedding_dimension': 1536, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
127
+ )
128
+ ```
129
+
130
+ ## Citing & Authors
131
+
132
+ The article has been presented at EPIA 2024 conference but the Springer proceedings are not available yet.
133
+ In the meantime, if you use this model you may cite the arXiv preprint:
134
+
135
+ @misc{gomes2024opensentenceembeddingsportuguese,
136
+ title={Open Sentence Embeddings for Portuguese with the Serafim PT* encoders family},
137
+ author={Lu铆s Gomes and Ant贸nio Branco and Jo茫o Silva and Jo茫o Rodrigues and Rodrigo Santos},
138
+ year={2024},
139
+ eprint={2407.19527},
140
+ archivePrefix={arXiv},
141
+ primaryClass={cs.CL},
142
+ url={https://arxiv.org/abs/2407.19527},
143
+ }
144
+
145
+