File size: 2,558 Bytes
14181f3 19c0041 a256709 19c0041 a256709 14181f3 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
---
license: cc-by-nc-4.0
language:
- en
pipeline_tag: zero-shot-image-classification
tags:
- medical
- multimodal
- vision-language pre-training
- chest x-ray
---
# MeDSLIP: Medical Dual-Stream Language-Image Pre-training for Fine-grained Alignment
## Introduction:
The official implementation code for "MeDSLIP: Medical Dual-Stream Language-Image Pre-training for Fine-grained Alignment".
[**Arxiv Version**](https://arxiv.org/abs/2403.10635)
## Quick Start:
Check checkpoints directory to download our pre-trained model from [Hugging Face: MeDSLIP](https://huggingface.co/pykale/MeDSLIP). It can be used for all zero-shot and finetuning tasks.
* **Zero-Shot Classification:**
We give an example on CXR14 in ```Sample_Zero-Shot_Classification_CXR14```. Change the data paths, and test our model by ```python test.py```.
We give an example on RSNA in ```Sample_Zero-Shot_Classification_RSNA```. Change the data paths, and test our model by ```python test.py```.
* **Zero-Shot Grounding:**
We give an example on RSNA_Pneumonia in ```Sample_Zero-Shot_Grounding_RSNA```. Change the data paths, and test our model by ```python test.py```.
* **Finetuning:**
We give segmentation and classification finetune code on SIIM_ACR dataset in ```Sample_Finetuning_SIIMACR```. Change the data paths, and finetune our model by ```python I1_classification/train_res_ft.py``` or ```python I2_segementation/train_res_ft.py```.
## Pre-train:
### Data Preparation
All files for data preparation files can be downloaded from [Hugging Face: MeDSLIP](https://huggingface.co/pykale/MeDSLIP).
- Extracted triplets: `landmark_observation_adj_mtx.npy`
- Training list: `train.json`
- Validation list: `valid.json`
- Test list: `test.json`
### Pre-training
Our pre-train code is given in ```PreTrain_MeDSLIP```.
* Check the ```PreTrain_MeDSLIP/data_file``` dir and download the files for data preparation.
* Change the data and preparation files paths as you disire in ```PreTrain_MeDSLIP/configs/Pretrain_MeDSLIP.yaml```, and ```python PreTrain_MeDSLIP/train_MeDSLIP.py``` to pre-train.
## Reference
```
@article{fan2024medslip,
title={MeDSLIP: Medical Dual-Stream Language-Image Pre-training for Fine-grained Alignment},
author={Fan, Wenrui and Suvon, Mohammod Naimul Islam and Zhou, Shuo and Liu, Xianyuan and Alabed, Samer and Osmani, Venet and Swift, Andrew and Chen, Chen and Lu, Haiping},
journal={arXiv preprint arXiv:2403.10635},
year={2024}
}
```
## Contact
If you have any question, please feel free to contact [email protected]. |