cambridgeltl/BioRedditBERT-uncased

BioRedditBERT

Model description

BioRedditBERT is a BERT model initialised from BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K) and further pre-trained on health-related Reddit posts. Please view our paper COMETA: A Corpus for Medical Entity Linking in the Social Media (EMNLP 2020) for more details.

Training data

We crawled all threads from 68 health themed subreddits such as r/AskDocs, r/health and etc. starting from the beginning of 2015 to the end of 2018, obtaining a collection of more than 800K discussions. This collection was then pruned by removing deleted posts, comments from bots or moderators, and so on. In the end, we obtained the training corpus with ca. 300 million tokens and a vocabulary size of ca. 780,000 words.

Training procedure

We use the same pre-training script in the original google-research/bert repo. The model is initialised with BioBERT-Base v1.0 + PubMed 200K + PMC 270K. We train with a batch size of 64, a max sequence length of 64, a learning rate of 2e-5 for 100k steps on two GeForce GTX 1080Ti (11 GB) GPUs. Other hyper-parameters are the same as default.

Eval results

To show the benefit from further pre-training on the social media domain, we demonstrate results on a medical entity linking dataset also in the social media: AskAPatient (Limsopatham and Collier 2016). We follow the same 10-fold cross-validation procedure for all models and report the average result without fine-tuning. [CLS] is used as representations for entity mentions (we also tried average of all tokens but found [CLS] generally performs better).

Model	Accuracy@1	Accuracy@5
BERT-base-uncased	38.2	43.3
BioBERT v1.1	41.4	51.5
ClinicalBERT	43.9	54.3
BlueBERT	41.5	48.5
SciBERT	42.3	51.9
PubMedBERT	42.5	49.6
BioRedditBERT	44.3	56.2

BibTeX entry and citation info

@inproceedings{basaldella-2020-cometa,
    title = "{COMETA}: A Corpus for Medical Entity Linking in the Social Media",
    author = "Basaldella, Marco  and Liu, Fangyu, and Shareghi, Ehsan, and Collier, Nigel",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2020",
    publisher = "Association for Computational Linguistics"
}