project-monai commited on
Commit
f7ea149
·
verified ·
1 Parent(s): f734d1a

Upload cxr_image_synthesis_latent_diffusion_model version 1.0.1

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ docs/figure_1.png filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,201 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ APPENDIX: How to apply the Apache License to your work.
179
+
180
+ To apply the Apache License to your work, attach the following
181
+ boilerplate notice, with the fields enclosed by brackets "[]"
182
+ replaced with your own identifying information. (Don't include
183
+ the brackets!) The text should be enclosed in the appropriate
184
+ comment syntax for the file format. We also recommend that a
185
+ file or class name and description of purpose be included on the
186
+ same "printed page" as the copyright notice for easier
187
+ identification within third-party archives.
188
+
189
+ Copyright [yyyy] [name of copyright owner]
190
+
191
+ Licensed under the Apache License, Version 2.0 (the "License");
192
+ you may not use this file except in compliance with the License.
193
+ You may obtain a copy of the License at
194
+
195
+ http://www.apache.org/licenses/LICENSE-2.0
196
+
197
+ Unless required by applicable law or agreed to in writing, software
198
+ distributed under the License is distributed on an "AS IS" BASIS,
199
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
+ See the License for the specific language governing permissions and
201
+ limitations under the License.
configs/inference.json ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "imports": [
3
+ "$import torch",
4
+ "$from datetime import datetime",
5
+ "$from pathlib import Path",
6
+ "$from transformers import CLIPTextModel",
7
+ "$from transformers import CLIPTokenizer"
8
+ ],
9
+ "bundle_root": ".",
10
+ "dataset_dir": "",
11
+ "dataset": "",
12
+ "evaluator": "",
13
+ "inferer": "",
14
+ "load_old": 1,
15
+ "model_dir": "$@bundle_root + '/models'",
16
+ "output_dir": "$@bundle_root + '/output'",
17
+ "create_output_dir": "$Path(@output_dir).mkdir(exist_ok=True)",
18
+ "prompt": "Big right-sided pleural effusion",
19
+ "prompt_list": "$['', @prompt]",
20
+ "guidance_scale": 7.0,
21
+ "device": "$torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')",
22
+ "tokenizer": "$CLIPTokenizer.from_pretrained(\"stabilityai/stable-diffusion-2-1-base\", subfolder=\"tokenizer\")",
23
+ "text_encoder": "$CLIPTextModel.from_pretrained(\"stabilityai/stable-diffusion-2-1-base\", subfolder=\"text_encoder\")",
24
+ "tokenized_prompt": "$@tokenizer(@prompt_list, padding=\"max_length\", [email protected]_max_length, truncation=True,return_tensors=\"pt\")",
25
+ "prompt_embeds": "$@text_encoder(@tokenized_prompt.input_ids.squeeze(1))[0].to(@device)",
26
+ "out_file": "$datetime.now().strftime('sample_%H%M%S_%d%m%Y')",
27
+ "autoencoder_def": {
28
+ "_target_": "monai.networks.nets.AutoencoderKL",
29
+ "spatial_dims": 2,
30
+ "in_channels": 1,
31
+ "out_channels": 1,
32
+ "latent_channels": 3,
33
+ "channels": [
34
+ 64,
35
+ 128,
36
+ 128,
37
+ 128
38
+ ],
39
+ "num_res_blocks": 2,
40
+ "norm_num_groups": 32,
41
+ "norm_eps": 1e-06,
42
+ "attention_levels": [
43
+ false,
44
+ false,
45
+ false,
46
+ false
47
+ ],
48
+ "with_encoder_nonlocal_attn": false,
49
+ "with_decoder_nonlocal_attn": false
50
+ },
51
+ "network_def": "@diffusion_def",
52
+ "load_autoencoder_path": "$@model_dir + '/autoencoder.pt'",
53
+ "load_autoencoder_func": "$@autoencoder_def.load_old_state_dict if bool(@load_old) else @autoencoder_def.load_state_dict",
54
+ "load_autoencoder": "$@load_autoencoder_func(torch.load(@load_autoencoder_path))",
55
+ "autoencoder": "$@autoencoder_def.to(@device)",
56
+ "diffusion_def": {
57
+ "_target_": "monai.networks.nets.DiffusionModelUNet",
58
+ "spatial_dims": 2,
59
+ "in_channels": 3,
60
+ "out_channels": 3,
61
+ "channels": [
62
+ 256,
63
+ 512,
64
+ 768
65
+ ],
66
+ "num_res_blocks": 2,
67
+ "attention_levels": [
68
+ false,
69
+ true,
70
+ true
71
+ ],
72
+ "norm_num_groups": 32,
73
+ "norm_eps": 1e-06,
74
+ "resblock_updown": false,
75
+ "num_head_channels": [
76
+ 0,
77
+ 512,
78
+ 768
79
+ ],
80
+ "with_conditioning": true,
81
+ "transformer_num_layers": 1,
82
+ "cross_attention_dim": 1024
83
+ },
84
+ "load_diffusion_path": "$@model_dir + '/model.pt'",
85
+ "load_diffusion_func": "$@diffusion_def.load_old_state_dict if bool(@load_old) else @diffusion_def.load_state_dict",
86
+ "load_diffusion": "$@load_diffusion_func(torch.load(@load_diffusion_path))",
87
+ "diffusion": "$@diffusion_def.to(@device)",
88
+ "scheduler": {
89
+ "_target_": "monai.networks.schedulers.DDIMScheduler",
90
+ "_requires_": [
91
+ "@load_diffusion",
92
+ "@load_autoencoder"
93
+ ],
94
+ "beta_start": 0.0015,
95
+ "beta_end": 0.0205,
96
+ "num_train_timesteps": 1000,
97
+ "schedule": "scaled_linear_beta",
98
+ "prediction_type": "v_prediction",
99
+ "clip_sample": false
100
+ },
101
+ "noise": "$torch.randn((1, 3, 64, 64)).to(@device)",
102
+ "set_timesteps": "[email protected]_timesteps(num_inference_steps=50)",
103
+ "sampler": {
104
+ "_target_": "scripts.sampler.Sampler",
105
+ "_requires_": "@set_timesteps"
106
+ },
107
+ "sample": "[email protected]_fn(@noise, @autoencoder, @diffusion, @scheduler, @prompt_embeds)",
108
+ "saver": {
109
+ "_target_": "scripts.saver.JPGSaver",
110
+ "_requires_": "@create_output_dir",
111
+ "output_dir": "@output_dir"
112
+ },
113
+ "run": "[email protected](@sample, @out_file)",
114
+ "save": "$torch.save(@sample, @output_dir + '/' + @out_file + '.pt')"
115
+ }
configs/logging.conf ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [loggers]
2
+ keys=root
3
+
4
+ [handlers]
5
+ keys=consoleHandler
6
+
7
+ [formatters]
8
+ keys=fullFormatter
9
+
10
+ [logger_root]
11
+ level=INFO
12
+ handlers=consoleHandler
13
+
14
+ [handler_consoleHandler]
15
+ class=StreamHandler
16
+ level=INFO
17
+ formatter=fullFormatter
18
+ args=(sys.stdout,)
19
+
20
+ [formatter_fullFormatter]
21
+ format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
configs/metadata.json ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_20240725.json",
3
+ "version": "1.0.1",
4
+ "changelog": {
5
+ "1.0.1": "update to huggingface hosting",
6
+ "1.0.0": "Initial release"
7
+ },
8
+ "monai_version": "1.4.0",
9
+ "pytorch_version": "2.5.1",
10
+ "numpy_version": "1.26.4",
11
+ "required_packages_version": {
12
+ "transformers": "4.46.3"
13
+ },
14
+ "task": "Chest X-ray image synthesis",
15
+ "description": "A generative model for creating high-resolution chest X-ray based on MIMIC dataset",
16
+ "copyright": "Copyright (c) MONAI Consortium",
17
+ "authors": "Walter Hugo Lopez Pinaya, Mark Graham, Eric Kerfoot, Virginia Fernandez",
18
+ "data_source": "https://physionet.org/content/mimic-cxr-jpg/2.0.0/",
19
+ "data_type": "image",
20
+ "image_classes": "Radiography (X-ray) with 512 x 512 pixels",
21
+ "intended_use": "This is a research tool/prototype and not to be used clinically",
22
+ "network_data_format": {
23
+ "inputs": {
24
+ "latent_representation": {
25
+ "type": "image",
26
+ "format": "magnitude",
27
+ "modality": "CXR",
28
+ "num_channels": 3,
29
+ "spatial_shape": [
30
+ 77,
31
+ 64,
32
+ 64
33
+ ],
34
+ "dtype": "float32",
35
+ "value_range": [],
36
+ "is_patch_data": false
37
+ },
38
+ "timesteps": {
39
+ "format": "magnitude",
40
+ "num_channels": 1,
41
+ "spatial_shape": [
42
+ 1
43
+ ],
44
+ "type": "vector",
45
+ "value_range": [
46
+ 0,
47
+ 1000
48
+ ],
49
+ "dtype": "long"
50
+ },
51
+ "context": {
52
+ "format": "magnitude",
53
+ "num_channels": 1024,
54
+ "spatial_shape": [
55
+ 1
56
+ ],
57
+ "type": "vector",
58
+ "value_range": [],
59
+ "dtype": "float32"
60
+ }
61
+ },
62
+ "outputs": {
63
+ "pred": {
64
+ "type": "image",
65
+ "format": "magnitude",
66
+ "modality": "CXR",
67
+ "num_channels": 1,
68
+ "spatial_shape": [
69
+ 512,
70
+ 512
71
+ ],
72
+ "dtype": "float32",
73
+ "value_range": [
74
+ 0,
75
+ 1
76
+ ],
77
+ "is_patch_data": false,
78
+ "channel_def": {
79
+ "0": "X-ray"
80
+ }
81
+ }
82
+ }
83
+ }
84
+ }
docs/README.md ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Description
2
+
3
+ A diffusion model to synthetise X-Ray images based on radiological report impressions.
4
+
5
+ # Model Overview
6
+ This model is trained from scratch using the Latent Diffusion Model architecture [1] and is used for the synthesis of
7
+ 2D Chest X-ray conditioned on Radiological reports. The model is divided into two parts: an autoencoder with a
8
+ KL-regularisation model that compresses data into a latent space and a diffusion model that learns to generate
9
+ conditioned synthetic latent representations. This model is conditioned on Findings and Impressions from radiological
10
+ reports. The original repository can be found [here](https://github.com/Warvito/generative_chestxray)
11
+
12
+ ![](./figure_1.png) <br>
13
+ <p align="center">
14
+ Figure 1 - Synthetic images from the model. </p>
15
+
16
+ # Data
17
+ The model was trained on brain data from 90,000 participants from the MIMIC dataset [2] [3]. We downsampled the
18
+ original images to have a format of 512 x 512 pixels.
19
+
20
+ ## Preprocessing
21
+ We resized the original images to make the smallest sides have 512 pixels. When inputting it to the network, we center
22
+ cropped the images to 512 x 512. The pixel intensity was normalised to be between [0, 1]. The text data was obtained
23
+ from associated radiological reports. We randoomly extracted sentences from the findings and impressions sections of the
24
+ reports, having a maximum of 5 sentences and 77 tokens. The text was tokenised using the CLIPTokenizer from
25
+ transformers package (https://github.com/huggingface/transformers) (pretrained model
26
+ "stabilityai/stable-diffusion-2-1-base") and then encoded using CLIPTextModel from the same package and pretrained
27
+ model.
28
+
29
+ # Examples of inference
30
+
31
+ Here we included a few examples of commands to sample images from the model and save them as .jpg files. The available
32
+ arguments for this task are: "--prompt" (str) text prompt to condition the model on; "--guidance_scale" (float), the
33
+ parameter that controls how much the image generation process follows the text prompt. The higher the value, the more
34
+ the image sticks to a given text input (the common range is between 1-21).
35
+
36
+ Examples:
37
+
38
+ ```shell
39
+ $ python -m monai.bundle run --config_file configs/inference.json --prompt "Big right-sided pleural effusion" --guidance_scale 7.0
40
+ ```
41
+
42
+ ```shell
43
+ $ python -m monai.bundle run --config_file configs/inference.json --prompt "Small right-sided pleural effusion" --guidance_scale 7.0
44
+ ```
45
+
46
+ ```shell
47
+ $ python -m monai.bundle run --config_file configs/inference.json --prompt "Bilateral pleural effusion" --guidance_scale 7.0
48
+ ```
49
+
50
+ ```shell
51
+ $ python -m monai.bundle run --config_file configs/inference.json --prompt "Cardiomegaly" --guidance_scale 7.0
52
+ ```
53
+
54
+ ## Using a new version of the model
55
+
56
+ If you want to use the checkpoints from a newly fine-tuned model, you need to set parameter load_old to 0 when you run inference,
57
+ to avoid the function load_old_state_dict being called instead of load_state_dict to be called, currently default, as it is
58
+ required to load the checkpoint from the original GenerativeModels repository.
59
+
60
+ ```shell
61
+ $ python -m monai.bundle run --config_file configs/inference.json --prompt "Pleural effusion." --guidance_scale 7.0 --load_old 0
62
+ ```
63
+
64
+ ## References
65
+
66
+
67
+ [1] Pinaya, Walter HL, et al. "Brain imaging generation with latent diffusion models." MICCAI Workshop on Deep Generative Models. Springer, Cham, 2022.
68
+
69
+ [2] Johnson, A., Lungren, M., Peng, Y., Lu, Z., Mark, R., Berkowitz, S., & Horng, S. (2019). MIMIC-CXR-JPG - chest radiographs with structured labels (version 2.0.0). PhysioNet. https://doi.org/10.13026/8360-t248.
70
+
71
+ [3] Johnson AE, Pollard TJ, Berkowitz S, Greenbaum NR, Lungren MP, Deng CY, Mark RG, Horng S. MIMIC-CXR: A large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042. 2019 Jan 21.
docs/figure_1.png ADDED

Git LFS Details

  • SHA256: 0231bcb80c716ffc01a58b5d0c5a9d384a1d0c79138c707f4b0562dafa58d5c4
  • Pointer size: 131 Bytes
  • Size of remote file: 478 kB
models/autoencoder.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:17a704d96984ce07423d98695949f19f91ebca58c562067603995a22368cb116
3
+ size 18491405
models/model.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3d3a16c4798847614fd262e4132c44414aa30b0aa04044ab1b2c4e23e73d03c3
3
+ size 1054758129
scripts/__init__.py ADDED
File without changes
scripts/sampler.py ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import torch
4
+ import torch.nn as nn
5
+ from monai.utils import optional_import
6
+ from torch.cuda.amp import autocast
7
+
8
+ tqdm, has_tqdm = optional_import("tqdm", name="tqdm")
9
+
10
+
11
+ class Sampler:
12
+ def __init__(self) -> None:
13
+ super().__init__()
14
+
15
+ @torch.no_grad()
16
+ def sampling_fn(
17
+ self,
18
+ noise: torch.Tensor,
19
+ autoencoder_model: nn.Module,
20
+ diffusion_model: nn.Module,
21
+ scheduler: nn.Module,
22
+ prompt_embeds: torch.Tensor,
23
+ guidance_scale: float = 7.0,
24
+ scale_factor: float = 0.3,
25
+ ) -> torch.Tensor:
26
+ if has_tqdm:
27
+ progress_bar = tqdm(scheduler.timesteps)
28
+ else:
29
+ progress_bar = iter(scheduler.timesteps)
30
+
31
+ for t in progress_bar:
32
+ noise_input = torch.cat([noise] * 2)
33
+ model_output = diffusion_model(
34
+ noise_input, timesteps=torch.Tensor((t,)).to(noise.device).long(), context=prompt_embeds
35
+ )
36
+ noise_pred_uncond, noise_pred_text = model_output.chunk(2)
37
+ noise_pred = noise_pred_uncond + guidance_scale * (noise_pred_text - noise_pred_uncond)
38
+ noise, _ = scheduler.step(noise_pred, t, noise)
39
+
40
+ with autocast():
41
+ sample = autoencoder_model.decode_stage_2_outputs(noise / scale_factor)
42
+
43
+ return sample
scripts/saver.py ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import numpy as np
4
+ import torch
5
+ from PIL import Image
6
+
7
+
8
+ class JPGSaver:
9
+ def __init__(self, output_dir: str) -> None:
10
+ super().__init__()
11
+ self.output_dir = output_dir
12
+
13
+ def save(self, image_data: torch.Tensor, file_name: str) -> None:
14
+ image_data = np.clip(image_data.cpu().numpy(), 0, 1)
15
+ image_data = (image_data * 255).astype(np.uint8)
16
+ im = Image.fromarray(image_data[0, 0])
17
+ im.save(self.output_dir + "/" + file_name + ".jpg")