Update README.md
Browse files
README.md
CHANGED
@@ -12,14 +12,14 @@ We release four variants of SciMult here:
|
|
12 |
**scimult_moe_pmcpatients_par.ckpt**
|
13 |
**scimult_moe_pmcpatients_ppr.ckpt**
|
14 |
|
15 |
-
**scimult_vanilla.ckpt** and **scimult_moe.ckpt** can be used for various scientific literature understanding tasks
|
16 |
|
17 |
**scimult_moe_pmcpatients_par.ckpt** and **scimult_moe_pmcpatients_ppr.ckpt** are initialized from **scimult_moe.ckpt** and continuously pre-trained on the training sets of [PMC-Patients](https://github.com/pmc-patients/pmc-patients) patient-to-article retrieval and patient-to-patient retrieval tasks, respectively. As of October 2023, these two models rank 1st in their corresponding tasks on the [PMC-Patients Leaderboard](https://pmc-patients.github.io/).
|
18 |
|
19 |
|
20 |
## Pre-training Data
|
21 |
SciMult is pre-trained on the following data:
|
22 |
-
[MAPLE](https://
|
23 |
[Citation Prediction Triplets](https://huggingface.co/datasets/allenai/scirepeval/viewer/cite_prediction) for link prediction
|
24 |
[SciRepEval-Search](https://huggingface.co/datasets/allenai/scirepeval/viewer/search) for literature retrieval
|
25 |
|
|
|
12 |
**scimult_moe_pmcpatients_par.ckpt**
|
13 |
**scimult_moe_pmcpatients_ppr.ckpt**
|
14 |
|
15 |
+
**scimult_vanilla.ckpt** and **scimult_moe.ckpt** can be used for various scientific literature understanding tasks. Their difference is that **scimult_vanilla.ckpt** adopts a typical 12-layer Transformer architecture (i.e., the same as [BERT base](https://huggingface.co/bert-base-uncased)), whereas **scimult_moe.ckpt** adopts a Mixture-of-Experts Transformer architecture with task-specific multi-head attention (MHA) sublayers. Experimental results show that **scimult_moe.ckpt** achieves better performance in general.
|
16 |
|
17 |
**scimult_moe_pmcpatients_par.ckpt** and **scimult_moe_pmcpatients_ppr.ckpt** are initialized from **scimult_moe.ckpt** and continuously pre-trained on the training sets of [PMC-Patients](https://github.com/pmc-patients/pmc-patients) patient-to-article retrieval and patient-to-patient retrieval tasks, respectively. As of October 2023, these two models rank 1st in their corresponding tasks on the [PMC-Patients Leaderboard](https://pmc-patients.github.io/).
|
18 |
|
19 |
|
20 |
## Pre-training Data
|
21 |
SciMult is pre-trained on the following data:
|
22 |
+
[MAPLE](https://zenodo.org/records/7611544) for paper classification
|
23 |
[Citation Prediction Triplets](https://huggingface.co/datasets/allenai/scirepeval/viewer/cite_prediction) for link prediction
|
24 |
[SciRepEval-Search](https://huggingface.co/datasets/allenai/scirepeval/viewer/search) for literature retrieval
|
25 |
|