|
--- |
|
language: |
|
- en |
|
metrics: |
|
- dice-score |
|
- area-under-the-receiver-operating-characteristic |
|
pipeline_tag: video-classification |
|
tags: |
|
- medical |
|
- deep-learning |
|
datasets: |
|
- arcade-dataset |
|
- mhi-clinical-database |
|
library_name: pytorch |
|
model-index: |
|
- name: deepcoro-segmentation-algorithm |
|
results: |
|
- task: |
|
type: image-segmentation |
|
dataset: |
|
name: arcade-dataset |
|
type: arcade-dataset |
|
metrics: |
|
- name: dice-score |
|
type: area-under-the-receiver-operating-characteristic |
|
value: 72.93 |
|
source: |
|
name: npj Digital Medicine |
|
url: https://www.nature.com/articles/s41746-024-01134-4 |
|
- name: deepcoro-stenosis-prediction-algorithm |
|
results: |
|
- task: |
|
type: video-classification |
|
dataset: |
|
name: mhi-clinical-database |
|
type: mhi-clinical-database |
|
metrics: |
|
- name: area-under-the-receiver-operating-characteristic |
|
type: area-under-the-receiver-operating-characteristic |
|
value: 0.8294 |
|
source: |
|
name: npj Digital Medicine |
|
url: https://www.nature.com/articles/s41746-024-01134-4 |
|
--- |
|
# Model Card for DeepCoro |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
DeepCoro is a deep learning model designed to run inference on DICOM coronary angiography videos. The model identifies stenoses, their locations, associated coronary artery segments, and their associetated percentage. This model repository contains the weights for DeepCoro's segmentation algorithm (Algorithm 4) and stenosis percentage prediction algorithm (Algorithm 6). Models for the primary anatomic struction classification (Algorithm 1) and stenosis detection (Algorithm 2) could not be made public. |
|
|
|
- **Developed by:** HeartWise AI |
|
- **Model type:** Deep learning supervised model for coronary angiography analysis |
|
- **Language(s) (NLP):** Not applicable |
|
- **License:** [See LICENSE agreement](https://github.com/HeartWise-AI/DeepCoro/tree/main?tab=License-1-ov-file) |
|
- **Finetuned from model:** Algorithm 6 was fine-tuned from [Torchvision's Swin3D base model](https://pytorch.org/vision/main/models/generated/torchvision.models.video.swin3d_b.html) |
|
|
|
### Model Sources |
|
|
|
- **Repository:** [DeepCoro GitHub](https://github.com/HeartWise-AI/DeepCoro) |
|
- **Paper:** [Evaluation of Stenoses Using AI Video Models Applied to Coronary Angiographies](https://www.nature.com/articles/s41746-024-01134-4) |
|
- **Demo:** [Supplementary Video 1 in the Supplementary Information of the paper](https://static-content.springer.com/esm/art%3A10.1038%2Fs41746-024-01134-4/MediaObjects/41746_2024_1134_MOESM2_ESM.mp4) |
|
|
|
## Uses |
|
|
|
### Direct Use |
|
|
|
DeepCoro is only suitable for use on DICOM format coronary angiographies to predict stenosis percentage, which meet the following requirements: |
|
* Contain information about: |
|
- Imager Pixel Spacing |
|
- Distance Source To Detector |
|
- Distance Source To Patient |
|
- Recommended Display Frame Rate |
|
- Study Date |
|
- Patient's Birth Date |
|
* Display the right coronary arteries (RCA) or the left coronary arteries (LCA). |
|
* Be aquired at 15 frames per seconds. |
|
|
|
### Downstream Use |
|
|
|
DeepCoro’s adaptability was successfully demonstrated by fine-tuning the model on quantitative coronary angiography (QCA) data, which further reduced the mean absolute error in stenosis percentage prediction. This capability suggests that the model can be extended to other quantitative assessment tasks in coronary angiography imaging. |
|
|
|
### Out-of-Scope Use |
|
|
|
The model is not suitable for use on non-DICOM formats data or videos that do not meet the specified requirements. The model is not intended for use outside of clinical settings or by individuals without appropriate medical training. It should not be used for diagnosing other conditions unrelated to coronary artery stenosis. |
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
|
DeepCoro's primary limitation is that the stenosis percentages for training and testing are based on clinician interpretation from CAG videos, which may not always align with actual stenosis values. Additionally, DeepCoro is optimized for detecting stenoses in 11 specific coronary segments, excluding side branches and CABG patients. The current average processing time of 62.60 seconds per DICOM video may pose a challenge in clinical settings where effectiveness is crucial. Moreover, running inference with the DeepCoro repository requires manual identification of coronary arteries and stenoses in an input CSV file, introducing potential human error. The model is in its development phase, and therefore should not be used for official diagnois. |
|
|
|
### Recommendations |
|
|
|
Users should be aware of the risks, biases, and limitations of the model. To mitigate the limitations identified in the DeepCoro model, it is recommended to exclude videos involving PCI and patients with previous CABG to enhance model precision. Users may calibrate the model using QCA assessments to further refine accuracy and reduce variability. Fine-tuning the model on diverse datasets can improve its versatility and reliability. For further recommendations, refer to the repository documentation. |
|
|
|
## How to Get Started with the Model |
|
|
|
Use the code below to run inference with DeepCoro: |
|
|
|
1. Clone the DeepCoro repository: |
|
```sh |
|
git clone https://github.com/HeartWise-AI/DeepCoro.git |
|
``` |
|
2. Place your input files in the `dcm_input/` folder. |
|
3. Build the Docker environment: |
|
```sh |
|
docker build -t deepcoro_inference . |
|
``` |
|
4. Run inference: |
|
- On CPU: |
|
```sh |
|
docker run -v /path/to/dcm_input:/dcm_input -v /path/to/results:/results deepcoro_inference |
|
``` |
|
- On GPU: |
|
```sh |
|
docker run --gpus all -v /path/to/dcm_input:/dcm_input -v /path/to/results:/results deepcoro_inference |
|
``` |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
The training data for Algorithm 4 was derived from the [ARCADE dataset](https://zenodo.org/records/8386059), which included X-ray coronary angiography images and their associated manual segmentation for each coronary artery segment. This dataset comprised 1200 images, covering a wide range of anatomical variations and stenosis conditions in the LCA and RCA. The ARCADE dataset provided a rich source of training material, with the training set making up approximately 66% of the total dataset. Data augmentation techniques were applied to enhance the model’s robustness and ability to generalize. |
|
|
|
For Algorithm 6, the training data came from the Montreal Heart Institute (MHI) database which contained 182,418 coronary angiography videos and their associated percentage of stenosis, visually assessed by cardiologists for every coronary artery segment. From this large dataset, 44,139 stenosis videos from 8057 patients were derived for training and validation purposes. The training dataset included approximately 75% of these videos, offering a diverse range of cases, from mild to severe stenosis, ensuring a comprehensive training experience. This dataset was crucial for developing the model’s ability to accurately detect and quantify stenosis across different coronary artery segments. |
|
|
|
### Training Procedure |
|
|
|
Algorithm 4 was developed through extensive testing over 200 epochs, involving nine state-of-the-art segmentation models: UNet, UNet++, MANet, LinkNet, FPN, PSPNet, DeepLabV3, DeepLabV3+, and PAN. The training process explored a variety of loss functions, including Jaccard loss, Dice loss, focal loss, Lovasz loss, and Tversky loss. Hyperparameter optimization was conducted using random search, considering batch sizes of 4, 8, 16, 32, and 64, and learning rates ranging from 1e−2 to 1e−5. The optimization process employed 5-fold cross-validation, ensuring consistent evaluation by using the same folds across all sets of hyperparameters. |
|
|
|
The training procedure for Algorithm 6 involved fine-tuning the Swin3D model, originally pre-trained on Kinetics400 and ImageNetV1. The model was adapted for regression tasks by modifying the activation function to Sigmoid and adjusting the output size. The training included a learning rate schedule, starting at 0.001 and adjusting based on the validation set performance, using the mean square error loss function and stochastic gradient descent optimizer. Hyperparameter tuning for Algorithm 6 was performed using learning rates of 0.00001, 0.0001, and 0.001 to minimize the validation loss. Batch sizes were chosen to maximize memory usage on the available GPU. |
|
|
|
### Preprocessing |
|
|
|
For preprocessing, stenosis videos are resized to 17.5mm x 17.5mm and registered. This involved aligning each frame with a reference frame to maintain the segment of interest's alignment across all frames within each stenosis box video. Before inputting the videos into Algorithm 6, they were resized to 224x224 pixels with interpolation and 24 frames centered around the reference area were used. |
|
|
|
### Training Hyperparameters |
|
|
|
Final hyperparameters for Algorithm 6 were a learning rate of 0.001 and a batch size of 4, chosen to maximize memory usage on the GPU. |
|
|
|
### Seepds, Sizes, Times |
|
|
|
The end-to-end processing time for DeepCoro, from inputting the video to obtaining the final analysis, is approximately 62.60 seconds in average. The sizes of the avalable model weights for DeepCoro are : |
|
- **Algorithm 4** (7 models): 104, 92.8, 104, 92.8, 86.1, 90 and 86.1 MB |
|
- **Algorithm 6** (1 model): 731 MB |
|
|
|
|
|
## Evaluation |
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
#### Testing Data |
|
|
|
The testing dataset for Algorithm 4, also from the [ARCADE dataset](https://zenodo.org/records/8386059), consisted of 200 images, representing 17% of the total dataset. This testing set was separate from the training set and included a diverse array of images. The testing data provided a balanced representation of different anatomical structures and conditions, allowing for an accurate assessment of the model’s generalization performance. |
|
|
|
The testing data for Algorithm 6 was obtained from the MHI database. A subset of 5904 videos (15% of the total dataset) was used for testing, ensuring a representative evaluation of the model’s performance in real-world scenarios. This testing set included a wide range of stenosis severities and anatomical variations, providing a comprehensive assessment of the model’s ability to generalize beyond the training data. |
|
|
|
#### Factors |
|
|
|
For Algorithm 4, metrics were disaggregated by coronary artery segment. Evaluation metrics for Algorithm 6 were disaggregated by coronary artery (LCA or RCA). |
|
|
|
#### Metrics |
|
|
|
For Aglorithm 4, evaluation metrics included the Dice score and the predictive positive value (PPV). For Algorithm 6, evaluation metrics included mean absolute error (MAE) for regression, and the area under the teceiver operating curve (AUROC) for classification. |
|
|
|
### Results |
|
|
|
The results for Algorithm 4, focused on segmentation, are evaluated using the Dice score and PPV across different coronary artery segments. These metrics provide insights into the accuracy and reliability of the segmentation model in delineating the anatomical structures of interest. The segmentation algorithm achieved notable performance, with variations across different segments, reflecting the inherent challenges in segmenting complex anatomical structures. |
|
|
|
The following table presents the Dice score and PPV for each coronary artery segment: |
|
|
|
|Coronary Artery Segment|Dice Score (%)|PPV (%)| |
|
|-------------------------|------------|---------| |
|
|left main artery|77.84|86.48| |
|
|proximal left anterior descending artery|70.89|68.70| |
|
|middle left anterior descending artery|74.20|76.53| |
|
|distal left anterior descending artery|71.40|74.13| |
|
|proximal left circumflex artery|75.14|74.35| |
|
|distal left circumflex artery|60.39|61.15| |
|
|proximal right coronary artery|80.33|80.92| |
|
|middle right coronary artery|72.58|78.87| |
|
|distal right coronary artery|77.26|76.92| |
|
|posterolateral branch from the right coronary artery|63.08|82.44| |
|
|posterior descending artery|74.11|77.02| |
|
|weighted average|72.93|75.96| |
|
|
|
|
|
**Algorithm 6: Stenosis Detection** |
|
|
|
For Algorithm 6, the evaluation was based on MAE and AUROC for the RCA, LCA, and combined RCA+LCA artery. These metrics highlight the algorithm's effectiveness in detecting and quantifying stenosis severity. |
|
|
|
The MAE and AUROC values for the stenosis detection algorithm are summarized in the table below: |
|
|
|
|Coronary Artery|MAE (%)|AUROC| |
|
|-------------------------|-----|-------| |
|
|RCA|17.82|0.8643| |
|
|LCA|22.19|0.8017| |
|
|RCA+LCA|20.15|0.8294| |
|
|
|
## Environmental Impact |
|
|
|
To train and test our Algorithm 4 and 6 models, we used: |
|
- **Hardware Type:** RTX3090 |
|
- **Hours used:** 200h |
|
- **Cloud Provider:** Private |
|
- **Compute Region:** Canada |
|
- **Carbon Emitted:** 2.1 kg of CO2 (computed with [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute)) |
|
|
|
## Technical Specifications |
|
|
|
### Model Architecture and Objective |
|
|
|
DeepCoro is an advanced AI-based pipeline designed for automatic evaluation of coronary angiography videos, focusing on the severity assessment of stenoses in the RCA and LCA. The pipeline integrates six specialized algorithms: |
|
|
|
1. Primary Anatomic Structure Detection Algorithm: Uses an Xception model to identify primary anatomic structures in CAG videos. |
|
2. Stenosis Detection Algorithm: Utilizes the RetinaNet architecture to locate coronary artery segments and stenoses. |
|
3. Registration Algorithm: Aligns stenosis bounding boxes across video frames, accounting for cardiac motion. |
|
4. Segmentation Algorithm: Applies an ensemble of seven models to segment videos, identifying 11 epicardial coronary artery segments. |
|
5. Coronary Artery Stenosis Assignment to Segment Algorithm: Assigns detected stenosis to specific coronary artery segments. |
|
6. Stenosis Percentage Prediction Algorithm: Uses a modified Swin3D transformer model to predict the severity of stenoses in percentage terms. |
|
|
|
### Compute Infrastructure |
|
|
|
Training and inference were conducted on high-performance GPUs with extensive memory and computational capabilities. |
|
|
|
#### Hardware |
|
|
|
High-performance RTX3090 GPUs with a 48GB memory. |
|
|
|
#### Software |
|
|
|
The model was developed using PyTorch and other supporting libraries for video processing and deep learning. |
|
|
|
## Citation |
|
|
|
**BibTeX:** |
|
```bibtex |
|
@article{avram2023evaluation, |
|
title={Evaluation of Stenoses Using AI Video Models Applied to Coronary Angiographies}, |
|
author={Labrecque Langlais, E. and Corbin, D. and Tastet, O. and Hayek, A. and Doolub, G. and Mrad, S. and Tardif, J.-C. and Tanguay, J.-F. and Marquis-Gravel, G. and Tison, G. and Kadoury, S. and Le, W. and Gallo, R. and Lesage, F. and Avram, R.}, |
|
year={2023} |
|
} |
|
``` |
|
|
|
**APA:** |
|
Labrecque Langlais, E., Corbin, D., Tastet, O., Hayek, A., Doolub, G., Mrad, S., Tardif, J.-C., Tanguay, J.-F., Marquis-Gravel, G., Tison, G., Kadoury, S., Le, W., Gallo, R., Lesage, F., & Avram, R. (2023). Evaluation of Stenoses Using AI Video Models Applied to Coronary Angiographies. PREPRINT (Version 1) available at Research Square. https://doi.org/10.21203/rs.3.rs-3610879/v1 |
|
|
|
## Glossary |
|
|
|
- **AUROC:** Area Under the Receiver Operating Characteristic Curve, a performance measurement for classification problems at various threshold settings. |
|
- **CABG**: Coronary Artery Bypass Grafting, a surgical procedure to restore normal blood flow to an obstructed coronary artery. |
|
- **CAG:** Coronary Angiography, a procedure that uses a special dye and X-rays to see how blood flows through the coronary arteries. |
|
- **Dice score**: A statistical tool to gauge the similarity between two sets of data, often used to measure the accuracy of image segmentation models. |
|
- **DICOM:** Digital Imaging and Communications in Medicine, a standard for handling, storing, printing, and transmitting information in medical imaging. |
|
- **LCA:** Left Coronary Artery, teh other main artery that supplies blood to the heart. |
|
- **MAE:** Mean Absolute Error, a measure of errors between paired continuous observations to describe regression performance. |
|
- **PCI**: Percutaneous Coronary Intervention, a non-surgical procedure used to treat narrowing of the coronary arteries of the heart found in coronary artery disease. |
|
- **PPV**: Predictive Positive Value, a measure used to assess the precision of a model, indicating the proportion of positive results that are true positives. |
|
- **QCA**: Quantitative Coronary Angiography, a method to quantitatively measure coronary artery dimensions and the degree of stenosis. |
|
- **RCA:** Right Coronary Artery, one of the two main arteries that supply blood to the heart. |
|
- **Stenosis:** The narrowing of blood vessels. |
|
|
|
## More Information |
|
|
|
For more details, refer to the published paper and supplementary materials. |
|
|
|
## Model Card Authors |
|
|
|
Élodie Labrecque Langlais, Denis Corbin, Olivier Tastet, Ahmad Hayek, Gemina Doolub, Sebastián Mrad, Jean-Claude Tardif, Jean-François Tanguay, Guillaume Marquis-Gravel, Geoffrey H. Tison, Samuel Kadoury, William Le, Richard Gallo, Frederic Lesage, Robert Avram |
|
|
|
## Model Card Contact |
|
|
|
[email protected] |
|
|
|
### Reference |
|
------------- |
|
Labrecque Langlais, E., Corbin, D., Tastet, O., Hayek, A., Doolub, G., Mrad, S., Tardif, J.-C., Tanguay, J.-F., Marquis-Gravel, G., Tison, G., Kadoury, S., Le, W., Gallo, R., Lesage, F, & Avram, R. Evaluation of Stenoses Using AI Video Models Applied to Coronary Angiographies, 23 November 2023, PREPRINT (Version 1) available at Research Square [https://doi.org/10.21203/rs.3.rs-3610879/v1] |
|
|
|
``` |
|
@article{avram2023evaluation, |
|
title={Evaluation of Stenoses Using AI Video Models Applied to Coronary Angiographies}, |
|
author={Labrecque Langlais, E. and Corbin, D. and Tastet, O. and Hayek, A. and Doolub, G. and Mrad, S. and Tardif, J.-C. and Tanguay, J.-F. and Marquis-Gravel, G. and Tison, G. and Kadoury, S. and Le, W. and Gallo, R. and Lesage, F. and Avram, R.}, |
|
year={2023} |
|
} |
|
``` |
