DRAGON Longformer large domain-specific

Pretrained model on Dutch clinical reports using a masked language modeling (MLM) objective. It was introduced in this paper. The model was pretrained using domain-specific data (i.e., clinical reports) from scratch. The architecture is the same as allenai/longformer-large-4096 from HuggingFace. The tokenizer was fitted to the dataset of Dutch medical reports, using the same settings for the tokenizer as roberta-base.

Model description

Longformer is a transformers model that was pretrained on a large corpus of Dutch clinical reports in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labeling them in any way with an automatic process to generate inputs and labels from those texts.

This way, the model learns an inner representation of the Dutch medical language that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled reports, for instance, you can train a standard classifier using the features produced by the BERT model as inputs.

Model variations

Multiple architectures were pretrained for the DRAGON challenge.

Model	#params	Language
`joeranbosma/dragon-bert-base-mixed-domain`	109M	Dutch → Dutch
`joeranbosma/dragon-roberta-base-mixed-domain`	278M	Multiple → Dutch
`joeranbosma/dragon-roberta-large-mixed-domain`	560M	Multiple → Dutch
`joeranbosma/dragon-longformer-base-mixed-domain`	149M	English → Dutch
`joeranbosma/dragon-longformer-large-mixed-domain`	435M	English → Dutch
`joeranbosma/dragon-bert-base-domain-specific`	109M	Dutch
`joeranbosma/dragon-roberta-base-domain-specific`	278M	Dutch
`joeranbosma/dragon-roberta-large-domain-specific`	560M	Dutch
`joeranbosma/dragon-longformer-base-domain-specific`	149M	Dutch
`joeranbosma/dragon-longformer-large-domain-specific`	435M	Dutch

Intended uses & limitations

You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.

Note that this model is primarily aimed at being fine-tuned on tasks that use the whole text (e.g., a clinical report) to make decisions, such as sequence classification, token classification or question answering. For tasks such as text generation you should look at model like GPT2.

How to use

You can use this model directly with a pipeline for masked language modeling:

from transformers import pipeline
unmasker = pipeline("fill-mask", model="joeranbosma/dragon-longformer-large-domain-specific")
unmasker("Dit onderzoek geen aanwijzingen voor significant carcinoom. PIRADS <mask>.")

Here is how to use this model to get the features of a given text in PyTorch:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("joeranbosma/dragon-longformer-large-domain-specific")
model = AutoModel.from_pretrained("joeranbosma/dragon-longformer-large-domain-specific")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors="pt")
output = model(**encoded_input)

Limitations and bias

Even if the training data used for this model could be characterized as fairly neutral, this model can have biased predictions. This bias will also affect all fine-tuned versions of this model.

Training data

For pretraining, 4,333,201 clinical reports (466,351 consecutive patients) were selected from Ziekenhuisgroep Twente from patients with a diagnostic or interventional visit between 13 July 2000 and 25 April 2023. 180,439 duplicate clinical reports (179,808 patients) were excluded, resulting in 4,152,762 included reports (463,692 patients). These reports were split into training (80%, 3,322,209 reports), validation (10%, 415,276 reports), and testing (10%, 415,277 reports). The testing reports were set aside for future analysis and are not used for pretraining.

Training procedure

Pretraining

The model was pretrained using masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then runs the entire masked sentence through the model and has to predict the masked words. This is different from traditional recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like GPT which internally masks the future tokens. It allows the model to learn a bidirectional representation of the sentence.

The details of the masking procedure for each sentence are the following:

15% of the tokens are masked.
In 80% of the cases, the masked tokens are replaced by [MASK].
In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
In the 10% remaining cases, the masked tokens are left as is.

The HuggingFace implementation was used for pretraining: run_mlm.py.

Pretraining hyperparameters

The following hyperparameters were used during pretraining:

learning_rate: 1e-4
train_batch_size: 4
eval_batch_size: 4
seed: 42
gradient_accumulation_steps: 64
total_train_batch_size: 256
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 10.0
max_seq_length: 4096

Framework versions

Transformers 4.29.0.dev0
Pytorch 2.0.0+cu117
Datasets 2.11.0
Tokenizers 0.13.3

Evaluation results

Pending evaluation on the DRAGON benchmark.

BibTeX entry and citation info

@article{PENDING}