add non-deterministic note

README.md

@@ -6,23 +6,20 @@ library_name: monai
 license: apache-2.0
 ---
 # Model Overview
-
 A pre-trained model for simultaneous segmentation and classification of nuclei within multi-tissue histology images based on CoNSeP data. The details of the model can be found in [1].
 
 ## Workflow
+The model is trained to simultaneously segment and classify nuclei. Training is done via a two-stage approach. First initialized the model with pre-trained weights on the [ImageNet dataset](https://ieeexplore.ieee.org/document/5206848), trained only the decoders for the first 50 epochs, and then fine-tuned all layers for another 50 epochs. There are two training modes in total. If "original" mode is specified, [270, 270] and [80, 80] are used for `patch_size` and `out_size` respectively. If "fast" mode is specified, [256, 256] and [164, 164] are used for `patch_size` and `out_size` respectively. The results shown below are based on the "fast" mode.
 
-The model is trained to simultaneous segment and classify nuclei. Training is done via a two-stage approach. First initialized the model with pre-trained weights on the [ImageNet dataset](https://ieeexplore.ieee.org/document/5206848), trained only the decoders for the first 50 epochs, and then fine-tuned all layers for another 50 epochs. There are two training modes in total. If "original" mode is specified, it uses [270, 270] and [80, 80] for `patch_size` and `out_size` respectively. If "fast" mode is specified, it uses [256, 256] and [164, 164] for `patch_size` and `out_size` respectively. The results we show below are based on the "fast" model.
-
-- We train the first stage with pre-trained weights from some internal data.
-
-- The original author's repo also has pre-trained weights which is for non-commercial use. Each user is responsible for checking the content of models/datasets and the applicable licenses and determining if suitable for the intended use. The license for the pre-trained model is different than MONAI license. Please check the source where these weights are obtained from: <https://github.com/vqdang/hover_net#data-format>
+### Pre-trained weights
+The first stage is trained with pre-trained weights from some internal data.The [original author's repo](https://github.com/vqdang/hover_net#data-format) also provides pre-trained weights but for non-commercial use.
+Each user is responsible for checking the content of models/datasets and the applicable licenses and determining if suitable for the intended use.
 
 `PRETRAIN_MODEL_URL` is "https://drive.google.com/u/1/uc?id=1KntZge40tAHgyXmHYVqZZ5d2p_4Qr2l5&export=download" which can be used in bash code below.
 
 ![Model workflow](https://developer.download.nvidia.com/assets/Clara/Images/monai_hovernet_pipeline.png)
 
 ## Data
-
 The training data is from <https://warwick.ac.uk/fac/cross_fac/tia/data/hovernet/>.
 
 - Target: segment instance-level nuclei and classify the nuclei type
@@ -39,7 +36,6 @@ python scripts/prepare_patches.py -root your-concep-dataset-path
 ```
 
 ## Training configuration
-
 This model utilized a two-stage approach. The training was performed with the following:
 
 - GPU: At least 24GB of GPU memory.
@@ -50,19 +46,16 @@ This model utilized a two-stage approach. The training was performed with the fo
 - Loss: HoVerNetLoss
 
 ## Input
-
 Input: RGB images
 
 ## Output
-
 Output: a dictionary with the following keys:
 
 1. nucleus_prediction: predict whether or not a pixel belongs to the nuclei or background
 2. horizontal_vertical: predict the horizontal and vertical distances of nuclear pixels to their centres of mass
 3. type_prediction: predict the type of nucleus for each pixel
 
-## Model Performance
-
+## Performance
 The achieved metrics on the validation data are:
 
 Fast mode:
@@ -72,10 +65,10 @@ Fast mode:
 
 Note: Binary Dice is calculated based on the whole input. PQ and F1d were calculated from https://github.com/vqdang/hover_net#inference.
 
-This bundle is non-deterministic, for more details please refer to https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html#torch.use_deterministic_algorithms
+Please note that this bundle is non-deterministic because of the bilinear interpolation used in the network. Therefore, reproducing the training process may not get exactly the same performance.
+Please refer to https://pytorch.org/docs/stable/notes/randomness.html#reproducibility for more details about reproducibility.
 
 #### Training Loss and Dice
-
 stage1:
 ![A graph showing the training loss and the mean dice over 50 epochs in stage1](https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_segmentation_classification_train_stage0_v2.png)
 
@@ -83,7 +76,6 @@ stage2:
 ![A graph showing the training loss and the mean dice over 50 epochs in stage2](https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_segmentation_classification_train_stage1_v2.png)
 
 #### Validation Dice
-
 stage1:
 
 ![A graph showing the validation mean dice over 50 epochs in stage1](https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_segmentation_classification_val_stage0_v2.png)
@@ -92,54 +84,51 @@ stage2:
 
 ![A graph showing the validation mean dice over 50 epochs in stage2](https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_segmentation_classification_val_stage1_v2.png)
 
-## commands example
+## MONAI Bundle Commands
+In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle. The CLI supports flexible use cases, such as overriding configs at runtime and predefining arguments in a file.
 
-Execute training, the evaluation in the training were evaluated on patches:
+For more details usage instructions, visit the [MONAI Bundle Configuration Page](https://docs.monai.io/en/latest/config_syntax.html).
 
-- Run first stage
+#### Execute training, the evaluation in the training were evaluated on patches:
 
+- Run first stage
 ```
 python -m monai.bundle run --config_file configs/train.json --network_def#pretrained_url `PRETRAIN_MODEL_URL` --stage 0
 ```
 
 - Run second stage
-
 ```
 python -m monai.bundle run --config_file configs/train.json --network_def#freeze_encoder False --network_def#pretrained_url None --stage 1
 ```
 
-Override the `train` config to execute multi-GPU training:
+#### Override the `train` config to execute multi-GPU training:
 
 - Run first stage
-
 ```
 torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run --config_file "['configs/train.json','configs/multi_gpu_train.json']" --batch_size 8 --network_def#freeze_encoder True --network_def#pretrained_url `PRETRAIN_MODEL_URL --stage 0
 ```
 
 - Run second stage
-
 ```
 torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run --config_file "['configs/train.json','configs/multi_gpu_train.json']" --batch_size 4 --network_def#freeze_encoder False --network_def#pretrained_url None --stage 1
 ```
 
-Override the `train` config to execute evaluation with the trained model, here we evaluated dice from the whole input instead of the patches:
+#### Override the `train` config to execute evaluation with the trained model, here we evaluated dice from the whole input instead of the patches:
 
 ```
 python -m monai.bundle run --config_file "['configs/train.json','configs/evaluate.json']"
 ```
 
-### Execute inference
+#### Execute inference:
 
 ```
 python -m monai.bundle run --config_file configs/inference.json
 ```
 
 # Disclaimer
-
 This is an example, not to be used for diagnostic purposes.
 
 # References
-
 [1] Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, Nasir Rajpoot, Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images, Medical Image Analysis, 2019 https://doi.org/10.1016/j.media.2019.101563
 
 # License

configs/metadata.json

@@ -1,7 +1,8 @@
 {
     "schema": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_hovernet_20221124.json",
-    "version": "0.1.5",
+    "version": "0.1.6",
     "changelog": {
+        "0.1.6": "add non-deterministic note",
         "0.1.5": "update benchmark on A100",
         "0.1.4": "adapt to BundleWorkflow interface",
         "0.1.3": "add name tag",

docs/README.md

@@ -1,21 +1,18 @@
 # Model Overview
-
 A pre-trained model for simultaneous segmentation and classification of nuclei within multi-tissue histology images based on CoNSeP data. The details of the model can be found in [1].
 
 ## Workflow
+The model is trained to simultaneously segment and classify nuclei. Training is done via a two-stage approach. First initialized the model with pre-trained weights on the [ImageNet dataset](https://ieeexplore.ieee.org/document/5206848), trained only the decoders for the first 50 epochs, and then fine-tuned all layers for another 50 epochs. There are two training modes in total. If "original" mode is specified, [270, 270] and [80, 80] are used for `patch_size` and `out_size` respectively. If "fast" mode is specified, [256, 256] and [164, 164] are used for `patch_size` and `out_size` respectively. The results shown below are based on the "fast" mode.
 
-The model is trained to simultaneous segment and classify nuclei. Training is done via a two-stage approach. First initialized the model with pre-trained weights on the [ImageNet dataset](https://ieeexplore.ieee.org/document/5206848), trained only the decoders for the first 50 epochs, and then fine-tuned all layers for another 50 epochs. There are two training modes in total. If "original" mode is specified, it uses [270, 270] and [80, 80] for `patch_size` and `out_size` respectively. If "fast" mode is specified, it uses [256, 256] and [164, 164] for `patch_size` and `out_size` respectively. The results we show below are based on the "fast" model.
-
-- We train the first stage with pre-trained weights from some internal data.
-
-- The original author's repo also has pre-trained weights which is for non-commercial use. Each user is responsible for checking the content of models/datasets and the applicable licenses and determining if suitable for the intended use. The license for the pre-trained model is different than MONAI license. Please check the source where these weights are obtained from: <https://github.com/vqdang/hover_net#data-format>
+### Pre-trained weights
+The first stage is trained with pre-trained weights from some internal data.The [original author's repo](https://github.com/vqdang/hover_net#data-format) also provides pre-trained weights but for non-commercial use.
+Each user is responsible for checking the content of models/datasets and the applicable licenses and determining if suitable for the intended use.
 
 `PRETRAIN_MODEL_URL` is "https://drive.google.com/u/1/uc?id=1KntZge40tAHgyXmHYVqZZ5d2p_4Qr2l5&export=download" which can be used in bash code below.
 
 ![Model workflow](https://developer.download.nvidia.com/assets/Clara/Images/monai_hovernet_pipeline.png)
 
 ## Data
-
 The training data is from <https://warwick.ac.uk/fac/cross_fac/tia/data/hovernet/>.
 
 - Target: segment instance-level nuclei and classify the nuclei type
@@ -32,7 +29,6 @@ python scripts/prepare_patches.py -root your-concep-dataset-path
 ```
 
 ## Training configuration
-
 This model utilized a two-stage approach. The training was performed with the following:
 
 - GPU: At least 24GB of GPU memory.
@@ -43,19 +39,16 @@ This model utilized a two-stage approach. The training was performed with the fo
 - Loss: HoVerNetLoss
 
 ## Input
-
 Input: RGB images
 
 ## Output
-
 Output: a dictionary with the following keys:
 
 1. nucleus_prediction: predict whether or not a pixel belongs to the nuclei or background
 2. horizontal_vertical: predict the horizontal and vertical distances of nuclear pixels to their centres of mass
 3. type_prediction: predict the type of nucleus for each pixel
 
-## Model Performance
-
+## Performance
 The achieved metrics on the validation data are:
 
 Fast mode:
@@ -65,10 +58,10 @@ Fast mode:
 
 Note: Binary Dice is calculated based on the whole input. PQ and F1d were calculated from https://github.com/vqdang/hover_net#inference.
 
-This bundle is non-deterministic, for more details please refer to https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html#torch.use_deterministic_algorithms
+Please note that this bundle is non-deterministic because of the bilinear interpolation used in the network. Therefore, reproducing the training process may not get exactly the same performance.
+Please refer to https://pytorch.org/docs/stable/notes/randomness.html#reproducibility for more details about reproducibility.
 
 #### Training Loss and Dice
-
 stage1:
 ![A graph showing the training loss and the mean dice over 50 epochs in stage1](https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_segmentation_classification_train_stage0_v2.png)
 
@@ -76,7 +69,6 @@ stage2:
 ![A graph showing the training loss and the mean dice over 50 epochs in stage2](https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_segmentation_classification_train_stage1_v2.png)
 
 #### Validation Dice
-
 stage1:
 
 ![A graph showing the validation mean dice over 50 epochs in stage1](https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_segmentation_classification_val_stage0_v2.png)
@@ -85,54 +77,51 @@ stage2:
 
 ![A graph showing the validation mean dice over 50 epochs in stage2](https://developer.download.nvidia.com/assets/Clara/Images/monai_pathology_segmentation_classification_val_stage1_v2.png)
 
-## commands example
+## MONAI Bundle Commands
+In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle. The CLI supports flexible use cases, such as overriding configs at runtime and predefining arguments in a file.
 
-Execute training, the evaluation in the training were evaluated on patches:
+For more details usage instructions, visit the [MONAI Bundle Configuration Page](https://docs.monai.io/en/latest/config_syntax.html).
 
-- Run first stage
+#### Execute training, the evaluation in the training were evaluated on patches:
 
+- Run first stage
 ```
 python -m monai.bundle run --config_file configs/train.json --network_def#pretrained_url `PRETRAIN_MODEL_URL` --stage 0
 ```
 
 - Run second stage
-
 ```
 python -m monai.bundle run --config_file configs/train.json --network_def#freeze_encoder False --network_def#pretrained_url None --stage 1
 ```
 
-Override the `train` config to execute multi-GPU training:
+#### Override the `train` config to execute multi-GPU training:
 
 - Run first stage
-
 ```
 torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run --config_file "['configs/train.json','configs/multi_gpu_train.json']" --batch_size 8 --network_def#freeze_encoder True --network_def#pretrained_url `PRETRAIN_MODEL_URL --stage 0
 ```
 
 - Run second stage
-
 ```
 torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run --config_file "['configs/train.json','configs/multi_gpu_train.json']" --batch_size 4 --network_def#freeze_encoder False --network_def#pretrained_url None --stage 1
 ```
 
-Override the `train` config to execute evaluation with the trained model, here we evaluated dice from the whole input instead of the patches:
+#### Override the `train` config to execute evaluation with the trained model, here we evaluated dice from the whole input instead of the patches:
 
 ```
 python -m monai.bundle run --config_file "['configs/train.json','configs/evaluate.json']"
 ```
 
-### Execute inference
+#### Execute inference:
 
 ```
 python -m monai.bundle run --config_file configs/inference.json
 ```
 
 # Disclaimer
-
 This is an example, not to be used for diagnostic purposes.
 
 # References
-
 [1] Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, Nasir Rajpoot, Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images, Medical Image Analysis, 2019 https://doi.org/10.1016/j.media.2019.101563
 
 # License