ZhiyuanChen commited on
Commit
969e0cc
1 Parent(s): e5b755f

Upload folder using huggingface_hub

Browse files
README.md ADDED
@@ -0,0 +1,270 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: rna
3
+ tags:
4
+ - Biology
5
+ - RNA
6
+ license: agpl-3.0
7
+ datasets:
8
+ - multimolecule/rfam
9
+ library_name: multimolecule
10
+ pipeline_tag: fill-mask
11
+ mask_token: "<mask>"
12
+ widget:
13
+ - example_title: "microRNA-21"
14
+ text: "UAGC<mask>UAUCAGACUGAUGUUGA"
15
+ output:
16
+ - label: "U"
17
+ score: 0.2819758355617523
18
+ - label: "K"
19
+ score: 0.25282594561576843
20
+ - label: "G"
21
+ score: 0.22668947279453278
22
+ - label: "D"
23
+ score: 0.06814167648553848
24
+ - label: "W"
25
+ score: 0.03735977038741112
26
+ ---
27
+
28
+ # RNA-MSM
29
+
30
+ Pre-trained model on non-coding RNA (ncRNA) with multi (homologous) sequence alignment using a masked language modeling (MLM) objective.
31
+
32
+ ## Disclaimer
33
+
34
+ This is an UNOFFICIAL implementation of the [Multiple sequence alignment-based RNA language model and its application to structural inference](https://doi.org/10.1093/nar/gkad1031) by Yikun Zhang, Mei Lang, Jiuhong Jiang, Zhiqiang Gao, et al.
35
+
36
+ The OFFICIAL repository of RNA-MSM is at [yikunpku/RNA-MSM](https://github.com/yikunpku/RNA-MSM).
37
+
38
+ > [!CAUTION]
39
+ > The MultiMolecule team is aware of a potential risk in reproducing the results of RNA-MSM.
40
+ >
41
+ > The original implementation of RNA-MSM used a custom tokenizer that does not append `<eos>` token to the end of the input sequence in consistent to MSA Transformer.
42
+ > This should not affect the performance of the model in most cases, but it can lead to unexpected behavior in some cases.
43
+ >
44
+ > Please set `eos_token=None` explicitly in the tokenizer if you want the exact behavior of the original implementation.
45
+ >
46
+ > See more at [issue #10](https://github.com/yikunpku/RNA-MSM/issues/10)
47
+
48
+ > [!TIP]
49
+ > The MultiMolecule team has confirmed that the provided model and checkpoints are producing the same intermediate representations as the original implementation.
50
+
51
+ **The team releasing RNA-MSM did not write this model card for this model so this model card has been written by the MultiMolecule team.**
52
+
53
+ ## Model Details
54
+
55
+ RNA-MSM is a [bert](https://huggingface.co/google-bert/bert-base-uncased)-style model pre-trained on a large corpus of non-coding RNA sequences in a self-supervised fashion. This means that the model was trained on the raw nucleotides of RNA sequences only, with an automatic process to generate inputs and labels from those texts. Please refer to the [Training Details](#training-details) section for more information on the training process.
56
+
57
+ ### Model Specification
58
+
59
+ | Num Layers | Hidden Size | Num Heads | Intermediate Size | Num Parameters (M) | FLOPs (G) | MACs (G) | Max Num Tokens |
60
+ | ---------- | ----------- | --------- | ----------------- | ------------------ | --------- | -------- | -------------- |
61
+ | 10 | 768 | 12 | 3072 | 95.92 | 21.66 | 10.57 | 1024 |
62
+
63
+ ### Links
64
+
65
+ - **Code**: [multimolecule.rnamsm](https://github.com/DLS5-Omics/multimolecule/tree/master/multimolecule/models/rnamsm)
66
+ - **Weights**: [multimolecule/rnamsm](https://huggingface.co/multimolecule/rnamsm)
67
+ - **Data**: [Rfam](https://rfam.org)
68
+ - **Paper**: [Multiple sequence alignment-based RNA language model and its application to structural inference](https://doi.org/10.1093/nar/gkad1031)
69
+ - **Developed by**: Yikun Zhang, Mei Lang, Jiuhong Jiang, Zhiqiang Gao, Fan Xu, Thomas Litfin, Ke Chen, Jaswinder Singh, Xiansong Huang, Guoli Song, Yonghong Tian, Jian Zhan, Jie Chen, Yaoqi Zhou
70
+ - **Model type**: [BERT](https://huggingface.co/google-bert/bert-base-uncased) - [MSA](https://doi.org/10.1101/2021.02.12.430858)
71
+ - **Original Repository**: [https://github.com/yikunpku/RNA-MSM](https://github.com/yikunpku/RNA-MSM)
72
+
73
+ ## Usage
74
+
75
+ The model file depends on the [`multimolecule`](https://multimolecule.danling.org) library. You can install it using pip:
76
+
77
+ ```bash
78
+ pip install multimolecule
79
+ ```
80
+
81
+ ### Direct Use
82
+
83
+ You can use this model directly with a pipeline for masked language modeling:
84
+
85
+ ```python
86
+ >>> import multimolecule # you must import multimolecule to register models
87
+ >>> from transformers import pipeline
88
+ >>> unmasker = pipeline('fill-mask', model='multimolecule/rnamsm')
89
+ >>> unmasker("uagc<mask>uaucagacugauguuga")
90
+
91
+ [{'score': 0.2819758355617523,
92
+ 'token': 9,
93
+ 'token_str': 'U',
94
+ 'sequence': 'U A G C U U A U C A G A C U G A U G U U G A'},
95
+ {'score': 0.25282594561576843,
96
+ 'token': 15,
97
+ 'token_str': 'K',
98
+ 'sequence': 'U A G C K U A U C A G A C U G A U G U U G A'},
99
+ {'score': 0.22668947279453278,
100
+ 'token': 8,
101
+ 'token_str': 'G',
102
+ 'sequence': 'U A G C G U A U C A G A C U G A U G U U G A'},
103
+ {'score': 0.06814167648553848,
104
+ 'token': 18,
105
+ 'token_str': 'D',
106
+ 'sequence': 'U A G C D U A U C A G A C U G A U G U U G A'},
107
+ {'score': 0.03735977038741112,
108
+ 'token': 14,
109
+ 'token_str': 'W',
110
+ 'sequence': 'U A G C W U A U C A G A C U G A U G U U G A'}]
111
+ ```
112
+
113
+ ### Downstream Use
114
+
115
+ #### Extract Features
116
+
117
+ Here is how to use this model to get the features of a given sequence in PyTorch:
118
+
119
+ ```python
120
+ from multimolecule import RnaTokenizer, RnaMsmModel
121
+
122
+
123
+ tokenizer = RnaTokenizer.from_pretrained('multimolecule/rnamsm')
124
+ model = RnaMsmModel.from_pretrained('multimolecule/rnamsm')
125
+
126
+ text = "UAGCUUAUCAGACUGAUGUUGA"
127
+ input = tokenizer(text, return_tensors='pt')
128
+
129
+ output = model(**input)
130
+ ```
131
+
132
+ #### Sequence Classification / Regression
133
+
134
+ **Note**: This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for sequence classification or regression.
135
+
136
+ Here is how to use this model as backbone to fine-tune for a sequence-level task in PyTorch:
137
+
138
+ ```python
139
+ import torch
140
+ from multimolecule import RnaTokenizer, RnaMsmForSequencePrediction
141
+
142
+
143
+ tokenizer = RnaTokenizer.from_pretrained('multimolecule/rnamsm')
144
+ model = RnaMsmForSequencePrediction.from_pretrained('multimolecule/rnamsm')
145
+
146
+ text = "UAGCUUAUCAGACUGAUGUUGA"
147
+ input = tokenizer(text, return_tensors='pt')
148
+ label = torch.tensor([1])
149
+
150
+ output = model(**input, labels=label)
151
+ ```
152
+
153
+ #### Nucleotide Classification / Regression
154
+
155
+ **Note**: This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for nucleotide classification or regression.
156
+
157
+ Here is how to use this model as backbone to fine-tune for a nucleotide-level task in PyTorch:
158
+
159
+ ```python
160
+ import torch
161
+ from multimolecule import RnaTokenizer, RnaMsmForNucleotidePrediction
162
+
163
+
164
+ tokenizer = RnaTokenizer.from_pretrained('multimolecule/rnamsm')
165
+ model = RnaMsmForNucleotidPrediction.from_pretrained('multimolecule/rnamsm')
166
+
167
+ text = "UAGCUUAUCAGACUGAUGUUGA"
168
+ input = tokenizer(text, return_tensors='pt')
169
+ label = torch.randint(2, (len(text), ))
170
+
171
+ output = model(**input, labels=label)
172
+ ```
173
+
174
+ #### Contact Classification / Regression
175
+
176
+ **Note**: This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for contact classification or regression.
177
+
178
+ Here is how to use this model as backbone to fine-tune for a contact-level task in PyTorch:
179
+
180
+ ```python
181
+ import torch
182
+ from multimolecule import RnaTokenizer, RnaMsmForContactPrediction
183
+
184
+
185
+ tokenizer = RnaTokenizer.from_pretrained('multimolecule/rnamsm')
186
+ model = RnaMsmForContactPrediction.from_pretrained('multimolecule/rnamsm')
187
+
188
+ text = "UAGCUUAUCAGACUGAUGUUGA"
189
+ input = tokenizer(text, return_tensors='pt')
190
+ label = torch.randint(2, (len(text), len(text)))
191
+
192
+ output = model(**input, labels=label)
193
+ ```
194
+
195
+ ## Training Details
196
+
197
+ RNA-MSM used Masked Language Modeling (MLM) as the pre-training objective: taking a sequence, the model randomly masks 15% of the tokens in the input then runs the entire masked sentence through the model and has to predict the masked tokens. This is comparable to the Cloze task in language modeling.
198
+
199
+ ### Training Data
200
+
201
+ The RNA-MSM model was pre-trained on [Rfam](https://rfam.org).
202
+ The Rfam database is a collection of RNA sequence families of structural RNAs including non-coding RNA genes as well as cis-regulatory elements.
203
+ RNA-MSM used Rfam 14.7 which contains 4,069 RNA families.
204
+
205
+ To avoid potential overfitting in structural inference, RNA-MSM excluded families with experimentally determined structures, such as ribosomal RNAs, transfer RNAs, and small nuclear RNAs. The final dataset contains 3,932 RNA families. The median value for the number of MSA sequences for these families by RNAcmap3 is 2,184.
206
+
207
+ To increase the number of homologous sequences, RNA-MSM used an automatic pipeline, RNAcmap3, for homolog search and sequence alignment. RNAcmap3 is a pipeline that combines the BLAST-N, INFERNAL, Easel, RNAfold and evolutionary coupling tools to generate homologous sequences.
208
+
209
+ RNA-MSM preprocessed all tokens by replacing "T"s with "U"s and substituting "R", "Y", "K", "M", "S", "W", "B", "D", "H", "V", "N" with "X".
210
+
211
+ Note that [`RnaTokenizer`][multimolecule.RnaTokenizer] will convert "T"s to "U"s for you, you may disable this behaviour by passing `replace_T_with_U=False`. `RnaTokenizer` does not perform other substitutions.
212
+
213
+ ### Training Procedure
214
+
215
+ #### Preprocessing
216
+
217
+ RNA-MSM used masked language modeling (MLM) as the pre-training objective. The masking procedure is similar to the one used in BERT:
218
+
219
+ - 15% of the tokens are masked.
220
+ - In 80% of the cases, the masked tokens are replaced by `<mask>`.
221
+ - In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
222
+ - In the 10% remaining cases, the masked tokens are left as is.
223
+
224
+ #### PreTraining
225
+
226
+ The model was trained on 8 NVIDIA V100 GPUs with 32GiB memories.
227
+
228
+ - Learning rate: 3e-4
229
+ - Weight decay: 3e-4
230
+ - Optimizer: Adam
231
+ - Learning rate warm-up: 16,000 steps
232
+ - Epochs: 300
233
+ - Batch Size: 1
234
+ - Dropout: 0.1
235
+
236
+ ## Citation
237
+
238
+ **BibTeX**:
239
+
240
+ ```bibtex
241
+ @article{zhang2023multiple,
242
+ author = {Zhang, Yikun and Lang, Mei and Jiang, Jiuhong and Gao, Zhiqiang and Xu, Fan and Litfin, Thomas and Chen, Ke and Singh, Jaswinder and Huang, Xiansong and Song, Guoli and Tian, Yonghong and Zhan, Jian and Chen, Jie and Zhou, Yaoqi},
243
+ title = "{Multiple sequence alignment-based RNA language model and its application to structural inference}",
244
+ journal = {Nucleic Acids Research},
245
+ volume = {52},
246
+ number = {1},
247
+ pages = {e3-e3},
248
+ year = {2023},
249
+ month = {11},
250
+ abstract = "{Compared with proteins, DNA and RNA are more difficult languages to interpret because four-letter coded DNA/RNA sequences have less information content than 20-letter coded protein sequences. While BERT (Bidirectional Encoder Representations from Transformers)-like language models have been developed for RNA, they are ineffective at capturing the evolutionary information from homologous sequences because unlike proteins, RNA sequences are less conserved. Here, we have developed an unsupervised multiple sequence alignment-based RNA language model (RNA-MSM) by utilizing homologous sequences from an automatic pipeline, RNAcmap, as it can provide significantly more homologous sequences than manually annotated Rfam. We demonstrate that the resulting unsupervised, two-dimensional attention maps and one-dimensional embeddings from RNA-MSM contain structural information. In fact, they can be directly mapped with high accuracy to 2D base pairing probabilities and 1D solvent accessibilities, respectively. Further fine-tuning led to significantly improved performance on these two downstream tasks compared with existing state-of-the-art techniques including SPOT-RNA2 and RNAsnap2. By comparison, RNA-FM, a BERT-based RNA language model, performs worse than one-hot encoding with its embedding in base pair and solvent-accessible surface area prediction. We anticipate that the pre-trained RNA-MSM model can be fine-tuned on many other tasks related to RNA structure and function.}",
251
+ issn = {0305-1048},
252
+ doi = {10.1093/nar/gkad1031},
253
+ url = {https://doi.org/10.1093/nar/gkad1031},
254
+ eprint = {https://academic.oup.com/nar/article-pdf/52/1/e3/55443207/gkad1031.pdf},
255
+ }
256
+ ```
257
+
258
+ ## Contact
259
+
260
+ Please use GitHub issues of [MultiMolecule](https://github.com/DLS5-Omics/multimolecule/issues) for any questions or comments on the model card.
261
+
262
+ Please contact the authors of the [RNA-MSM paper](https://doi.org/10.1093/nar/gkad1031) for questions or comments on the paper/model.
263
+
264
+ ## License
265
+
266
+ This model is licensed under the [AGPL-3.0 License](https://www.gnu.org/licenses/agpl-3.0.html).
267
+
268
+ ```spdx
269
+ SPDX-License-Identifier: AGPL-3.0-or-later
270
+ ```
config.json ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "RnaMsmForPreTraining"
4
+ ],
5
+ "attention_bias": true,
6
+ "attention_dropout": 0.1,
7
+ "attention_type": "standard",
8
+ "bos_token_id": 1,
9
+ "embed_positions_msa": true,
10
+ "eos_token_id": 2,
11
+ "head": {
12
+ "act": null,
13
+ "bias": true,
14
+ "dropout": 0.0,
15
+ "hidden_size": null,
16
+ "layer_norm_eps": 1e-12,
17
+ "num_labels": null,
18
+ "output_name": null,
19
+ "problem_type": null,
20
+ "transform": null,
21
+ "transform_act": "gelu"
22
+ },
23
+ "hidden_act": "gelu",
24
+ "hidden_dropout": 0.1,
25
+ "hidden_size": 768,
26
+ "id2label": {
27
+ "0": "LABEL_0"
28
+ },
29
+ "initializer_range": 0.02,
30
+ "intermediate_size": 3072,
31
+ "label2id": {
32
+ "LABEL_0": 0
33
+ },
34
+ "layer_norm_eps": 1e-12,
35
+ "layer_type": "standard",
36
+ "lm_head": {
37
+ "act": null,
38
+ "bias": true,
39
+ "dropout": 0.0,
40
+ "hidden_size": 768,
41
+ "layer_norm_eps": 1e-12,
42
+ "output_name": null,
43
+ "transform": "nonlinear",
44
+ "transform_act": "gelu"
45
+ },
46
+ "mask_token_id": 4,
47
+ "max_position_embeddings": 1024,
48
+ "max_tokens_per_msa": 16384,
49
+ "model_type": "rnamsm",
50
+ "null_token_id": 5,
51
+ "num_attention_heads": 12,
52
+ "num_hidden_layers": 10,
53
+ "pad_token_id": 0,
54
+ "position_embedding_type": "absolute",
55
+ "torch_dtype": "float32",
56
+ "transformers_version": "4.44.0",
57
+ "unk_token_id": 3,
58
+ "use_cache": true,
59
+ "vocab_size": 26
60
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:15a0bce62e8d0b936a1727c6ef28f291fb1382732f49ad28503a2c5abf4e6e83
3
+ size 386173696
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a954fa91e6c0e4bc1a4b6b58baf399f28f63590ebb88e45a4baff1535cd87984
3
+ size 386233650
special_tokens_map.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<null>"
4
+ ],
5
+ "bos_token": "<cls>",
6
+ "cls_token": "<cls>",
7
+ "eos_token": "<eos>",
8
+ "mask_token": "<mask>",
9
+ "pad_token": "<pad>",
10
+ "sep_token": "<eos>",
11
+ "unk_token": "<unk>"
12
+ }
tokenizer_config.json ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "<pad>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "<cls>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "<eos>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "3": {
28
+ "content": "<unk>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "4": {
36
+ "content": "<mask>",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ },
43
+ "5": {
44
+ "content": "<null>",
45
+ "lstrip": false,
46
+ "normalized": false,
47
+ "rstrip": false,
48
+ "single_word": false,
49
+ "special": true
50
+ }
51
+ },
52
+ "additional_special_tokens": [
53
+ "<null>"
54
+ ],
55
+ "bos_token": "<cls>",
56
+ "clean_up_tokenization_spaces": true,
57
+ "cls_token": "<cls>",
58
+ "codon": false,
59
+ "eos_token": "<eos>",
60
+ "mask_token": "<mask>",
61
+ "model_max_length": 1024,
62
+ "nmers": 1,
63
+ "pad_token": "<pad>",
64
+ "replace_T_with_U": true,
65
+ "sep_token": "<eos>",
66
+ "tokenizer_class": "RnaTokenizer",
67
+ "unk_token": "<unk>"
68
+ }
vocab.txt ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <pad>
2
+ <cls>
3
+ <eos>
4
+ <unk>
5
+ <mask>
6
+ <null>
7
+ A
8
+ C
9
+ G
10
+ U
11
+ N
12
+ R
13
+ Y
14
+ S
15
+ W
16
+ K
17
+ M
18
+ B
19
+ D
20
+ H
21
+ V
22
+ .
23
+ X
24
+ *
25
+ -
26
+ I