|
|
--- |
|
|
library_name: transformers |
|
|
tags: |
|
|
- bulk RNA-seq |
|
|
- biology |
|
|
- transcriptomics |
|
|
--- |
|
|
|
|
|
# BulkRNABert |
|
|
|
|
|
BulkRNABert is a transformer-based, encoder-only language model pre-trained on bulk RNA-seq data using self-supervision via masked language modeling, following BERT’s method. It can be further fine-tuned for cancer type classification and survival time prediction on the TCGA dataset. |
|
|
|
|
|
**Developed by:** [InstaDeep](https://huggingface.co/InstaDeepAI) |
|
|
|
|
|
### Model Sources |
|
|
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
|
|
- [**Repository**](https://github.com/instadeepai/multiomics-open-research) |
|
|
- **Paper:** [BulkRNABert: Cancer prognosis from bulk RNA-seq based language models](https://proceedings.mlr.press/v259/gelard25a.html) |
|
|
|
|
|
### How to use |
|
|
|
|
|
Until its next release, the transformers library needs to be installed from source using the following command to use the models. |
|
|
PyTorch should also be installed. |
|
|
|
|
|
``` |
|
|
pip install --upgrade git+https://github.com/huggingface/transformers.git |
|
|
pip install torch |
|
|
``` |
|
|
|
|
|
A small snippet of code is provided below to run inference with the model using random input. |
|
|
|
|
|
``` |
|
|
import torch |
|
|
from transformers import AutoConfig, AutoModel |
|
|
|
|
|
model = AutoModel.from_pretrained( |
|
|
"InstaDeepAI/BulkRNABert", |
|
|
trust_remote_code=True, |
|
|
) |
|
|
|
|
|
n_genes = model.config.n_genes |
|
|
dummy_gene_expressions = torch.randint(0, model.config.n_expressions_bins, (1, n_genes)) |
|
|
torch_output = model(dummy_gene_expressions) |
|
|
``` |
|
|
|
|
|
A more complete example is provided in the repository. |
|
|
|
|
|
|
|
|
### Citing our work |
|
|
|
|
|
``` |
|
|
@InProceedings{pmlr-v259-gelard25a, |
|
|
title = {BulkRNABert: Cancer prognosis from bulk RNA-seq based language models}, |
|
|
author = {G{\'{e}}lard, Maxence and Richard, Guillaume and Pierrot, Thomas and Courn{\`{e}}de, Paul-Henry}, |
|
|
booktitle = {Proceedings of the 4th Machine Learning for Health Symposium}, |
|
|
pages = {384--400}, |
|
|
year = {2025}, |
|
|
editor = {Hegselmann, Stefan and Zhou, Helen and Healey, Elizabeth and Chang, Trenton and Ellington, Caleb and Mhasawade, Vishwali and Tonekaboni, Sana and Argaw, Peniel and Zhang, Haoran}, |
|
|
volume = {259}, |
|
|
series = {Proceedings of Machine Learning Research}, |
|
|
month = {15--16 Dec}, |
|
|
publisher = {PMLR}, |
|
|
url = {https://proceedings.mlr.press/v259/gelard25a.html}, |
|
|
} |
|
|
``` |
|
|
|
