license: apache-2.0
GatorTronS overview
Developed by a joint effort between the University of Florida and NVIDIA, GatorTronS is a clinical language model of 345 million parameters, pre-trained using a BERT architecure implemented in the Megatron package (https://github.com/NVIDIA/Megatron-LM).
GatorTronS is pre-trained using a dataset consisting of:
- 22B synthetic clinical words generated by GatorTronGPT (a Megatron GPT-3 model)
- 6.1B words from PubMed CC0,
- 2.5B words from WikiText,
- 0.5B words of de-identified clinical notes from MIMIC-III
The Github for GatorTronGPT is at : https://github.com/uf-hobi-informatics-lab/GatorTronGPT
This model is converted to Hugginface from : https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_s
22B synthetic clinical text description
We sampled the beginning 15 tokens from all sections of the de-identified notes from the MIMIC III database and generated approximately 8 million prompts. We also tried several random seeds in GatorTronGPT to generate multiple documents from one prompt. We controlled GatorTronGPT to generate a maximum length of 512 tokens. We apply GatorTronGPT to generate a total of 22 billion words of synthetic clinical text. Detailed information is provided in the GatorTronGPT paper: https://arxiv.org/abs/2305.13523
Model variations
| Model | Parameter |
|---|---|
| gatortron-base | 345 million |
| gatortronS (this model) | 345 million |
| gatortron-medium | 3.9 billion |
| gatortron-large | 8.9 billion |
How to use
from transformers import AutoModel, AutoTokenizer, AutoConfig
tokinizer= AutoTokenizer.from_pretrained('UFNLP/gatortronS')
config=AutoConfig.from_pretrained('UFNLP/gatortronS')
mymodel=AutoModel.from_pretrained('UFNLP/gatortronS')
encoded_input=tokinizer("Bone scan: Negative for distant metastasis.", return_tensors="pt")
encoded_output = mymodel(**encoded_input)
print (encoded_output)
- An NLP pacakge using GatorTronS for clinical concept extraction (Named Entity Recognition): https://github.com/uf-hobi-informatics-lab/ClinicalTransformerNER
- An NLP pacakge using GatorTronS for Relation Extraction: https://github.com/uf-hobi-informatics-lab/ClinicalTransformerRelationExtraction
- An NLP pacakge using GatorTronS for extraction of social determinants of health (SDoH) from clinical narratives: https://github.com/uf-hobi-informatics-lab/SDoH_SODA
Citation info
Peng C, Yang X, Chen A, Smith KE, PourNejatian N, Costa AB, Martin C, Flores MG, Zhang Y, Magoc T, Lipori G, Mitchell DA, Ospina NS, Ahmed MM, Hogan WR, Shenkman EA, Guo Y, Bian J, Wu Y†. A Study of Generative Large Language Model for Medical Research and Healthcare. 2023; https://arxiv.org/abs/2305.13523.
- BibTeX entry
@ARTICLE{Peng2023-sm,
title = "A study of generative large language model for medical
research and healthcare",
author = "Peng, Cheng and Yang, Xi and Chen, Aokun and Smith, Kaleb E
and PourNejatian, Nima and Costa, Anthony B and Martin,
Cheryl and Flores, Mona G and Zhang, Ying and Magoc, Tanja
and Lipori, Gloria and Mitchell, Duane A and Ospina, Naykky
S and Ahmed, Mustafa M and Hogan, William R and Shenkman,
Elizabeth A and Guo, Yi and Bian, Jiang and Wu, Yonghui",
month = may,
year = 2023,
copyright = "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
archivePrefix = "arXiv",
primaryClass = "cs.CL",
eprint = "2305.13523"
}
Contact
- Yonghui Wu: [email protected]
- Cheng Peng: [email protected]