Upload 9 files

CHANGELOG.md

@@ -0,0 +1,12 @@
+
+v1.0.1
+------
+
+- renamed main model `IF-I-IF` --> `IF-I-XL`
+- moved dir `notebooks` to HF storage https://huggingface.co/DeepFloyd/IF-I-XL-v1.0/tree/main/notebooks; lets keep new notebooks there;
+- added additional kaggle notebook (more free GPU resources) how to generate pictures 1k: [![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://www.kaggle.com/code/shonenkov/deepfloyd-if-4-3b-generator-of-pictures)
+
+v1.0.0
+------
+
+- initial version

LICENSE

@@ -0,0 +1,26 @@
+Copyright (c) 2023 DeepFloyd, StabilityAI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+1. The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+2. All persons obtaining a copy or substantial portion of the Software,
+a modified version of the Software (or substantial portion thereof), or
+a derivative work based upon this Software (or substantial portion thereof)
+must not delete, remove, disable, diminish, or circumvent any inference filters or
+inference filter mechanisms in the Software, or any portion of the Software that
+implements any such filters or filter mechanisms.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

LICENSE-MODEL

@@ -0,0 +1,175 @@
+DEEPFLOYD IF LICENSE AGREEMENT
+
+This License Agreement (as may be amended in accordance with this License Agreement, “License”),
+between you, or your employer or other entity (if you are entering into this agreement on behalf
+of your employer or other entity) (“Licensee” or “you”) and Stability AI Ltd.. (“Stability AI” or “we”)
+applies to your use of any computer program, algorithm, source code, object code, or software that is made
+available by Stability AI under this License (“Software”) and any specifications, manuals, documentation,
+and other written information provided by Stability AI related to the Software (“Documentation”).
+By clicking “I Accept” below or by using the Software, you agree to the terms of this License.
+If you do not agree to this License, then you do not have any rights to use the Software or
+Documentation (collectively, the “Software Products”), and you must immediately cease using
+the Software Products. If you are agreeing to be bound by the terms of this License on behalf
+of your employer or other entity, you represent and warrant to Stability AI that you have full legal
+authority to bind your employer or such entity to this License. If you do not have the requisite authority,
+you may not accept the License or access the Software Products on behalf of your employer or other entity.
+
+1. LICENSE GRANT
+
+a. Subject to your compliance with the Documentation and Sections 2, 3, and 5, Stability AI grants
+you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, royalty free and limited
+license under Stability AI’s copyright interests to reproduce, distribute, and create derivative works of
+the Software solely for your non-commercial research purposes. The foregoing license is personal to you,
+and you may not assign or sublicense this License or any other rights or obligations under this License
+without Stability AI’s prior written consent; any such assignment or sublicense will be void and will
+automatically and immediately terminate this License.
+
+b. You may make a reasonable number of copies of the Documentation solely for use in connection with
+the license to the Software granted above.
+
+c. The grant of rights expressly set forth in this Section 1 (License Grant) are the complete
+grant of rights to you in the Software Products, and no other licenses are granted, whether by waiver,
+estoppel, implication, equity or otherwise. Stability AI and its licensors reserve all rights
+not expressly granted by this License.
+
+
+2. RESTRICTIONS
+
+You will not, and will not permit, assist or cause any third party to:
+
+a. use, modify, copy, reproduce, create derivative works of, or distribute the Software Products
+(or any derivative works thereof, works incorporating the Software Products, or any data produced
+by the Software), in whole or in part, for (i) any commercial or production purposes,
+(ii) military purposes or in the service of nuclear technology, (iii) purposes of surveillance,
+including any research or development relating to surveillance, (iv) biometric processing,
+(v) in any manner that infringes, misappropriates, or otherwise violates any third-party rights,
+or (vi) in any manner that violates any applicable law and violating any privacy or security laws,
+rules, regulations, directives, or governmental requirements (including the General Data Privacy
+Regulation (Regulation (EU) 2016/679), the California Consumer Privacy Act, and any and all laws
+governing the processing of biometric information), as well as all amendments and successor laws
+to any of the foregoing;
+
+b. alter or remove copyright and other proprietary notices which appear on or in the Software Products;
+
+c. utilize any equipment, device, software, or other means to circumvent or remove any security or
+protection used by Stability AI in connection with the Software, or to circumvent or remove any
+usage restrictions, or to enable functionality disabled by Stability AI; or
+
+d. offer or impose any terms on the Software Products that alter, restrict, or are inconsistent
+with the terms of this License.
+
+e. 1) violate any applicable U.S. and non-U.S. export control and trade sanctions laws
+(“Export Laws”); 2) directly or indirectly export, re-export, provide, or otherwise
+transfer Software Products: (a) to any individual, entity, or country prohibited by Export Laws; (b)
+to anyone on U.S. or non-U.S. government restricted parties lists; or (c) for any purpose prohibited
+by Export Laws, including nuclear, chemical or biological weapons, or missile technology applications;
+3) use or download Software Products if you or they are: (a) located in a comprehensively sanctioned
+jurisdiction, (b) currently listed on any U.S. or non-U.S. restricted parties list, or (c) for any
+purpose prohibited by Export Laws; and (4) will not disguise your location through IP proxying or other methods.
+
+
+3. ATTRIBUTION
+
+Together with any copies of the Software Products (as well as derivative works thereof or works
+incorporating the Software Products) that you distribute, you must provide (i) a copy of this License,
+and (ii) the following attribution notice: “DeepFloyd is licensed under the DeepFloyd License,
+Copyright (c) Stability AI Ltd. All Rights Reserved.”
+
+
+4. DISCLAIMERS
+
+THE SOFTWARE PRODUCTS ARE PROVIDED “AS IS” and “WITH ALL FAULTS” WITH NO WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED. STABILITY AIEXPRESSLY DISCLAIMS ALL REPRESENTATIONS AND WARRANTIES, EXPRESS OR IMPLIED,
+WHETHER BY STATUTE, CUSTOM, USAGE OR OTHERWISE AS TO ANY MATTERS RELATED TO THE SOFTWARE PRODUCTS,
+INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE,
+TITLE, SATISFACTORY QUALITY, OR NON-INFRINGEMENT. STABILITY AI MAKES NO WARRANTIES OR REPRESENTATIONS
+THAT THE SOFTWARE PRODUCTS WILL BE ERROR FREE OR FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS,
+OR PRODUCE ANY PARTICULAR RESULTS.
+
+
+5. LIMITATION OF LIABILITY
+
+TO THE FULLEST EXTENT PERMITTED BY LAW, IN NO EVENT WILL STABILITY AI BE LIABLE TO YOU (A) UNDER
+ANY THEORY OF LIABILITY, WHETHER BASED IN CONTRACT, TORT, NEGLIGENCE, STRICT LIABILITY, WARRANTY,
+OR OTHERWISE UNDER THIS LICENSE, OR (B) FOR ANY INDIRECT, CONSEQUENTIAL, EXEMPLARY, INCIDENTAL,
+PUNITIVE OR SPECIAL DAMAGES OR LOST PROFITS, EVEN IF STABILITY AI HAS BEEN ADVISED OF THE POSSIBILITY
+OF SUCH DAMAGES. THE SOFTWARE PRODUCTS, THEIR CONSTITUENT COMPONENTS, AND ANY OUTPUT
+(COLLECTIVELY, “SOFTWARE MATERIALS”) ARE NOT DESIGNED OR INTENDED FOR USE IN ANY APPLICATION OR
+SITUATION WHERE FAILURE OR FAULT OF THE SOFTWARE MATERIALS COULD REASONABLY BE ANTICIPATED TO LEAD
+TO SERIOUS INJURY OF ANY PERSON, INCLUDING POTENTIAL DISCRIMINATION OR VIOLATION OF AN INDIVIDUAL’S
+PRIVACY RIGHTS, OR TO SEVERE PHYSICAL, PROPERTY, OR ENVIRONMENTAL DAMAGE (EACH, A “HIGH-RISK USE”).
+IF YOU ELECT TO USE ANY OF THE SOFTWARE MATERIALS FOR A HIGH-RISK USE, YOU DO SO AT YOUR OWN RISK.
+YOU AGREE TO DESIGN AND IMPLEMENT APPROPRIATE DECISION-MAKING AND RISK-MITIGATION PROCEDURES AND
+POLICIES IN CONNECTION WITH A HIGH-RISK USE SUCH THAT EVEN IF THERE IS A FAILURE OR FAULT IN ANY
+OF THE SOFTWARE MATERIALS, THE SAFETY OF PERSONS OR PROPERTY AFFECTED BY THE ACTIVITY STAYS AT A LEVEL
+THAT IS REASONABLE, APPROPRIATE, AND LAWFUL FOR THE FIELD OF THE HIGH-RISK USE.
+
+
+6. INDEMNIFICATION
+
+You will indemnify, defend and hold harmless Stability AI and our subsidiaries and affiliates,
+and each of our respective shareholders, directors, officers, employees, agents, successors,
+and assigns (collectively, the “Stability AI Parties”) from and against any losses, liabilities,
+damages, fines, penalties, and expenses (including reasonable attorneys’ fees) incurred by any
+Stability AI Party in connection with any claim, demand, allegation, lawsuit, proceeding, or
+investigation (collectively, “Claims”) arising out of or related to: (a) your access to or
+use of the Software Products (as well as any results or data generated from such access or use),
+including any High-Risk Use (defined below); (b) your violation of this License; or (c)
+your violation, misappropriation or infringement of any rights of another (including intellectual
+property or other proprietary rights and privacy rights). You will promptly notify the Stability AI
+Parties of any such Claims, and cooperate with Stability AI Parties in defending such Claims.
+You will also grant the Stability AI Parties sole control of the defense or settlement,
+at Stability AI’s sole option, of any Claims. This indemnity is in addition to, and not in lieu of,
+any other indemnities or remedies set forth in a written agreement between you and
+Stability AI or the other Stability AI Parties.
+
+
+7. TERMINATION; SURVIVAL
+
+a. This License will automatically terminate upon any breach by you of the terms of this License.
+
+b. We may terminate this License, in whole or in part, at any time upon notice (including electronic) to you.
+
+c. The following sections survive termination of this License: 2 (Restrictions), 3 (Attribution),
+4 (Disclaimers), 5 (Limitation on Liability), 6 (Indemnification) 7 (Termination; Survival),
+8 (Third Party Materials), 9 (Trademarks), 10 (Applicable Law; Dispute Resolution), and 11 (Miscellaneous).
+
+
+8. THIRD PARTY MATERIALS
+
+The Software Products may contain third-party software or other components (including free and
+open source software) (all of the foregoing, “Third Party Materials”), which are subject to
+the license terms of the respective third-party licensors. Your dealings or correspondence
+with third parties and your use of or interaction with any Third Party Materials are solely
+between you and the third party. Stability AI does not control or endorse, and makes
+no representations or warranties regarding, any Third Party Materials, and your access
+to and use of such Third Party Materials are at your own risk.
+
+
+9. TRADEMARKS
+
+Licensee has not been granted any trademark license as part of this License and may not use any name
+or mark associated with Stability AI without the prior written permission of Stability AI, except to
+the extent necessary to make the reference required by the “ATTRIBUTION” section of this Agreement.
+
+
+10. APPLICABLE LAW; DISPUTE RESOLUTION
+
+This License will be governed and construed under the laws of the State of California without regard
+to conflicts of law provisions. Any suit or proceeding arising out of or relating to this License
+will be brought in the federal or state courts, as applicable, in San Mateo County, California,
+and each party irrevocably submits to the jurisdiction and venue of such courts.
+
+
+11. MISCELLANEOUS
+
+If any provision or part of a provision of this License is unlawful, void or unenforceable,
+that provision or part of the provision is deemed severed from this License, and will not affect
+the validity and enforceability of any remaining provisions. The failure of Stability AI to exercise
+or enforce any right or provision of this License will not operate as a waiver of such right or provision.
+This License does not confer any third-party beneficiary rights upon any other person or entity.
+This License, together with the Documentation, contains the entire understanding between you and
+Stability AI regarding the subject matter of this License, and supersedes all other written or
+oral agreements and understandings between you and Stability AI regarding such subject matter.
+No change or addition to any provision of this License will be binding unless it is in writing and
+signed by an authorized representative of both you and Stability AI.

README.md

@@ -1,11 +1,346 @@
----
-title: Ass
-emoji: 👁
-colorFrom: red
-colorTo: blue
-sdk: docker
-pinned: false
-license: deepfloyd-if-license
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+[![License](https://img.shields.io/badge/Code_License-Modified_MIT-blue.svg)](LICENSE)
+[![License](https://img.shields.io/badge/Weights_License-DeepFloyd_IF-orange.svg)](LICENSE-MODEL)
+[![Downloads](https://pepy.tech/badge/deepfloyd_if)](https://pepy.tech/project/deepfloyd_if)
+
+# IF by DeepFloyd Lab at [StabilityAI](https://stability.ai/)
+
+<p align="center">
+  <img src="./pics/nabla.jpg" width="100%">
+</p>
+
+We introduce DeepFloyd IF, a novel state-of-the-art open-source text-to-image model with a high degree of photorealism and language understanding. DeepFloyd IF is a modular composed of a frozen text encoder and three cascaded pixel diffusion modules: a base model that generates 64x64 px image based on text prompt and two super-resolution models, each designed to generate images of increasing resolution: 256x256 px and 1024x1024 px. All stages of the model utilize a frozen text encoder based on the T5 transformer to extract text embeddings, which are then fed into a UNet architecture enhanced with cross-attention and attention pooling. The result is a highly efficient model that outperforms current state-of-the-art models, achieving a zero-shot FID score of 6.66 on the COCO dataset. Our work underscores the potential of larger UNet architectures in the first stage of cascaded diffusion models and depicts a promising future for text-to-image synthesis.
+
+<p align="center">
+  <img src="./pics/deepfloyd_if_scheme.jpg" width="100%">
+</p>
+
+*Inspired by* [*Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding*](https://arxiv.org/pdf/2205.11487.pdf)
+
+## Minimum requirements to use all IF models:
+- 16GB vRAM for IF-I-XL (4.3B text to 64x64 base module) & IF-II-L (1.2B to 256x256 upscaler module)
+- 24GB vRAM for IF-I-XL (4.3B text to 64x64 base module) & IF-II-L (1.2B to 256x256 upscaler module) & Stable x4 (to 1024x1024 upscaler)
+- `xformers` and set env variable `FORCE_MEM_EFFICIENT_ATTN=1`
+
+
+## Quick Start
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/deepfloyd_if_free_tier_google_colab.ipynb)
+[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/DeepFloyd/IF)
+
+```shell
+pip install deepfloyd_if==1.0.1
+pip install xformers==0.0.16
+pip install git+https://github.com/openai/CLIP.git --no-deps
+```
+
+## Local notebooks
+[![Jupyter Notebook](https://img.shields.io/badge/jupyter_notebook-%23FF7A01.svg?logo=jupyter&logoColor=white)](https://huggingface.co/DeepFloyd/IF-I-XL-v1.0/blob/main/notebooks/pipes-DeepFloyd-IF-v1.0.ipynb)
+[![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://www.kaggle.com/code/shonenkov/deepfloyd-if-4-3b-generator-of-pictures)
+
+The Dream, Style Transfer, Super Resolution or Inpainting modes are avaliable in a Jupyter Notebook [here](https://huggingface.co/DeepFloyd/IF-I-XL-v1.0/blob/main/notebooks/pipes-DeepFloyd-IF-v1.0.ipynb).
+
+
+
+## Integration with 🤗 Diffusers
+
+IF is also integrated with the 🤗 Hugging Face [Diffusers library](https://github.com/huggingface/diffusers/).
+
+Diffusers runs each stage individually allowing the user to customize the image generation process as well as allowing to inspect intermediate results easily.
+
+### Example
+
+Before you can use IF, you need to accept its usage conditions. To do so:
+1. Make sure to have a [Hugging Face account](https://huggingface.co/join) and be loggin in
+2. Accept the license on the model card of [DeepFloyd/IF-I-XL-v1.0](https://huggingface.co/DeepFloyd/IF-I-XL-v1.0)
+3. Make sure to login locally. Install `huggingface_hub`
+```sh
+pip install huggingface_hub --upgrade
+```
+
+run the login function in a Python shell
+
+```py
+from huggingface_hub import login
+
+login()
+```
+
+and enter your [Hugging Face Hub access token](https://huggingface.co/docs/hub/security-tokens#what-are-user-access-tokens).
+
+Next we install `diffusers` and dependencies:
+
+```sh
+pip install diffusers accelerate transformers safetensors
+```
+
+And we can now run the model locally.
+
+By default `diffusers` makes use of [model cpu offloading](https://huggingface.co/docs/diffusers/optimization/fp16#model-offloading-for-fast-inference-and-memory-savings) to run the whole IF pipeline with as little as 14 GB of VRAM.
+
+If you are using `torch>=2.0.0`, make sure to **delete all** `enable_xformers_memory_efficient_attention()`
+functions.
+
+```py
+from diffusers import DiffusionPipeline
+from diffusers.utils import pt_to_pil
+import torch
+
+# stage 1
+stage_1 = DiffusionPipeline.from_pretrained("DeepFloyd/IF-I-XL-v1.0", variant="fp16", torch_dtype=torch.float16)
+stage_1.enable_xformers_memory_efficient_attention()  # remove line if torch.__version__ >= 2.0.0
+stage_1.enable_model_cpu_offload()
+
+# stage 2
+stage_2 = DiffusionPipeline.from_pretrained(
+    "DeepFloyd/IF-II-L-v1.0", text_encoder=None, variant="fp16", torch_dtype=torch.float16
+)
+stage_2.enable_xformers_memory_efficient_attention()  # remove line if torch.__version__ >= 2.0.0
+stage_2.enable_model_cpu_offload()
+
+# stage 3
+safety_modules = {"feature_extractor": stage_1.feature_extractor, "safety_checker": stage_1.safety_checker, "watermarker": stage_1.watermarker}
+stage_3 = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-x4-upscaler", **safety_modules, torch_dtype=torch.float16)
+stage_3.enable_xformers_memory_efficient_attention()  # remove line if torch.__version__ >= 2.0.0
+stage_3.enable_model_cpu_offload()
+
+prompt = 'a photo of a kangaroo wearing an orange hoodie and blue sunglasses standing in front of the eiffel tower holding a sign that says "very deep learning"'
+
+# text embeds
+prompt_embeds, negative_embeds = stage_1.encode_prompt(prompt)
+
+generator = torch.manual_seed(0)
+
+# stage 1
+image = stage_1(prompt_embeds=prompt_embeds, negative_prompt_embeds=negative_embeds, generator=generator, output_type="pt").images
+pt_to_pil(image)[0].save("./if_stage_I.png")
+
+# stage 2
+image = stage_2(
+    image=image, prompt_embeds=prompt_embeds, negative_prompt_embeds=negative_embeds, generator=generator, output_type="pt"
+).images
+pt_to_pil(image)[0].save("./if_stage_II.png")
+
+# stage 3
+image = stage_3(prompt=prompt, image=image, generator=generator, noise_level=100).images
+image[0].save("./if_stage_III.png")
+```
+
+ There are multiple ways to speed up the inference time and lower the memory consumption even more with `diffusers`. To do so, please have a look at the Diffusers docs:
+
+- 🚀 [Optimizing for inference time](https://huggingface.co/docs/diffusers/api/pipelines/if#optimizing-for-speed)
+- ⚙️ [Optimizing for low memory during inference](https://huggingface.co/docs/diffusers/api/pipelines/if#optimizing-for-memory)
+
+For more in-detail information about how to use IF, please have a look at [the IF blog post](https://huggingface.co/blog/if) and [the documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/if) 📖.
+
+## Run the code locally
+
+### Loading the models into VRAM
+
+```python
+from deepfloyd_if.modules import IFStageI, IFStageII, StableStageIII
+from deepfloyd_if.modules.t5 import T5Embedder
+
+device = 'cuda:0'
+if_I = IFStageI('IF-I-XL-v1.0', device=device)
+if_II = IFStageII('IF-II-L-v1.0', device=device)
+if_III = StableStageIII('stable-diffusion-x4-upscaler', device=device)
+t5 = T5Embedder(device="cpu")
+```
+
+### I. Dream
+Dream is the text-to-image mode of the IF model
+
+```python
+from deepfloyd_if.pipelines import dream
+
+prompt = 'ultra close-up color photo portrait of rainbow owl with deer horns in the woods'
+count = 4
+
+result = dream(
+    t5=t5, if_I=if_I, if_II=if_II, if_III=if_III,
+    prompt=[prompt]*count,
+    seed=42,
+    if_I_kwargs={
+        "guidance_scale": 7.0,
+        "sample_timestep_respacing": "smart100",
+    },
+    if_II_kwargs={
+        "guidance_scale": 4.0,
+        "sample_timestep_respacing": "smart50",
+    },
+    if_III_kwargs={
+        "guidance_scale": 9.0,
+        "noise_level": 20,
+        "sample_timestep_respacing": "75",
+    },
+)
+
+if_III.show(result['III'], size=14)
+```
+![](./pics/dream-III.jpg)
+
+## II. Zero-shot Image-to-Image Translation
+
+![](./pics/img_to_img_scheme.jpeg)
+
+In Style Transfer mode, the output of your prompt comes out at the style of the `support_pil_img`
+```python
+from deepfloyd_if.pipelines import style_transfer
+
+result = style_transfer(
+    t5=t5, if_I=if_I, if_II=if_II,
+    support_pil_img=raw_pil_image,
+    style_prompt=[
+        'in style of professional origami',
+        'in style of oil art, Tate modern',
+        'in style of plastic building bricks',
+        'in style of classic anime from 1990',
+    ],
+    seed=42,
+    if_I_kwargs={
+        "guidance_scale": 10.0,
+        "sample_timestep_respacing": "10,10,10,10,10,10,10,10,0,0",
+        'support_noise_less_qsample_steps': 5,
+    },
+    if_II_kwargs={
+        "guidance_scale": 4.0,
+        "sample_timestep_respacing": 'smart50',
+        "support_noise_less_qsample_steps": 5,
+    },
+)
+if_I.show(result['II'], 1, 20)
+```
+
+![Alternative Text](./pics/deep_floyd_if_image_2_image.gif)
+
+
+## III. Super Resolution
+For super-resolution, users can run `IF-II` and `IF-III` or 'Stable x4' on an image that was not necessarely generated by IF (two cascades):
+
+```python
+from deepfloyd_if.pipelines import super_resolution
+
+middle_res = super_resolution(
+    t5,
+    if_III=if_II,
+    prompt=['woman with a blue headscarf and a blue sweaterp, detailed picture, 4k dslr, best quality'],
+    support_pil_img=raw_pil_image,
+    img_scale=4.,
+    img_size=64,
+    if_III_kwargs={
+        'sample_timestep_respacing': 'smart100',
+        'aug_level': 0.5,
+        'guidance_scale': 6.0,
+    },
+)
+high_res = super_resolution(
+    t5,
+    if_III=if_III,
+    prompt=[''],
+    support_pil_img=middle_res['III'][0],
+    img_scale=4.,
+    img_size=256,
+    if_III_kwargs={
+        "guidance_scale": 9.0,
+        "noise_level": 20,
+        "sample_timestep_respacing": "75",
+    },
+)
+show_superres(raw_pil_image, high_res['III'][0])
+```
+
+![](./pics/if_as_upscaler.jpg)
+
+
+### IV. Zero-shot Inpainting
+
+```python
+from deepfloyd_if.pipelines import inpainting
+
+result = inpainting(
+    t5=t5, if_I=if_I,
+    if_II=if_II,
+    if_III=if_III,
+    support_pil_img=raw_pil_image,
+    inpainting_mask=inpainting_mask,
+    prompt=[
+        'oil art, a man in a hat',
+    ],
+    seed=42,
+    if_I_kwargs={
+        "guidance_scale": 7.0,
+        "sample_timestep_respacing": "10,10,10,10,10,0,0,0,0,0",
+        'support_noise_less_qsample_steps': 0,
+    },
+    if_II_kwargs={
+        "guidance_scale": 4.0,
+        'aug_level': 0.0,
+        "sample_timestep_respacing": '100',
+    },
+    if_III_kwargs={
+        "guidance_scale": 9.0,
+        "noise_level": 20,
+        "sample_timestep_respacing": "75",
+    },
+)
+if_I.show(result['I'], 2, 3)
+if_I.show(result['II'], 2, 6)
+if_I.show(result['III'], 2, 14)
+```
+![](./pics/deep_floyd_if_inpainting.gif)
+
+### 🤗 Model Zoo 🤗
+The link to download the weights as well as the model cards will be available soon on each model of the model zoo
+
+#### Original
+
+| Name                                                      | Cascade | Params | FID  | Batch size | Steps |
+|:----------------------------------------------------------|:-------:|:------:|:----:|:----------:|:-----:|
+| [IF-I-M](https://huggingface.co/DeepFloyd/IF-I-M-v1.0)    |    I    |  400M  | 8.86 |    3072    | 2.5M  |
+| [IF-I-L](https://huggingface.co/DeepFloyd/IF-I-L-v1.0)    |    I    |  900M  | 8.06 |    3200    | 3.0M  |
+| [IF-I-XL](https://huggingface.co/DeepFloyd/IF-I-XL-v1.0)* |    I    |  4.3B  | 6.66 |    3072    | 2.42M |
+| [IF-II-M](https://huggingface.co/DeepFloyd/IF-II-M-v1.0)  |   II    |  450M  |  -   |    1536    | 2.5M  |
+| [IF-II-L](https://huggingface.co/DeepFloyd/IF-II-L-v1.0)* |   II    |  1.2B  |  -   |    1536    | 2.5M  |
+| IF-III-L* _(soon)_                                        |   III   |  700M  |  -   |    3072    | 1.25M |
+
+ *best modules
+
+### Quantitative Evaluation
+
+`FID = 6.66`
+
+![](./pics/fid30k_if.jpg)
+
+## License
+
+The code in this repository is released under the bespoke license (see added [point two](https://github.com/deep-floyd/IF/blob/main/LICENSE#L13)).
+
+The weights will be available soon via [the DeepFloyd organization at Hugging Face](https://huggingface.co/DeepFloyd) and have their own LICENSE.
+
+**Disclaimer:** *The initial release of the IF model is under a restricted research-purposes-only license temporarily to gather feedback, and after that we intend to release a fully open-source model in line with other Stability AI models.*
+
+## Limitations and Biases
+
+The models available in this codebase have known limitations and biases. Please refer to [the model card](https://huggingface.co/DeepFloyd/IF-I-L-v1.0) for more information.
+
+
+## 🎓 DeepFloyd IF creators:
+- [Alex Shonenkov](https://github.com/shonenkov)
+- [Misha Konstantinov](https://github.com/zeroshot-ai)
+- [Daria Bakshandaeva](https://github.com/Gugutse)
+- [Christoph Schuhmann](https://github.com/christophschuhmann)
+- [Ksenia Ivanova](https://github.com/ivksu)
+- [Nadiia Klokova](https://github.com/vauimpuls)
+
+## 📄 Research Paper (Soon)
+
+## Acknowledgements
+
+Special thanks to [StabilityAI](http://stability.ai) and its CEO [Emad Mostaque](https://twitter.com/emostaque) for invaluable support, providing GPU compute and infrastructure to train the models (our gratitude goes to [Richard Vencu](https://github.com/rvencu)); thanks to [LAION](https://laion.ai) and [Christoph Schuhmann](https://github.com/christophschuhmann) in particular for contribution to the project and well-prepared datasets; thanks to [Huggingface](https://huggingface.co) teams for optimizing models' speed and memory consumption during inference, creating demos and giving cool advice!
+
+## 🚀 External Contributors 🚀
+- The Biggest Thanks [@Apolinário](https://github.com/apolinario), for ideas, consultations, help and support on all stages to make IF available in open-source; for writing a lot of documentation and instructions; for creating a friendly atmosphere in difficult moments 🦉;
+- Thanks, [@patrickvonplaten](https://github.com/patrickvonplaten), for improving loading time of unet models by 80%;
+for integration Stable-Diffusion-x4 as native pipeline 💪;
+- Thanks, [@williamberman](https://github.com/williamberman) and [@patrickvonplaten](https://github.com/patrickvonplaten) for diffusers integration 🙌;
+- Thanks, [@hysts](https://github.com/hysts) and [@Apolinário](https://github.com/apolinario) for creating [the best gradio demo with IF](https://huggingface.co/spaces/DeepFloyd/IF) 🚀;
+- Thanks, [@Dango233](https://github.com/Dango233), for adapting IF with xformers memory efficient attention 💪;

requirements-dev.txt

@@ -0,0 +1,2 @@
+-r requirements-test.txt
+pre-commit

requirements-test.txt

@@ -0,0 +1,3 @@
+-r requirements.txt
+pytest
+pytest-cov

requirements.txt

@@ -0,0 +1,15 @@
+tqdm
+numpy
+torch<2.0.0
+torchvision
+omegaconf
+matplotlib
+Pillow>=9.2.0
+huggingface_hub>=0.13.2
+transformers~=4.25.1
+accelerate~=0.15.0
+diffusers~=0.16.0
+tokenizers~=0.13.2
+sentencepiece~=0.1.97
+ftfy~=6.1.1
+beautifulsoup4~=4.11.1

setup.cfg

@@ -0,0 +1,13 @@
+[pep8]
+max-line-length = 120
+exclude = .tox,*migrations*,.json
+
+[flake8]
+max-line-length = 120
+exclude = .tox,*migrations*,.json
+
+[autopep8-wrapper]
+exclude = .tox,*migrations*,.json
+
+[check-docstring-first]
+exclude = .tox,*migrations*,.json

setup.py

@@ -0,0 +1,57 @@
+# -*- coding: utf-8 -*-
+import os
+import re
+from setuptools import setup
+
+
+def read(filename):
+    with open(os.path.join(os.path.dirname(__file__), filename)) as f:
+        file_content = f.read()
+    return file_content
+
+
+def get_requirements():
+    requirements = []
+    for requirement in read('requirements.txt').splitlines():
+        if requirement.startswith('git+') or requirement.startswith('svn+') or requirement.startswith('hg+'):
+            parsed_requires = re.findall(r'#egg=([\w\d\.]+)-([\d\.]+)$', requirement)
+            if parsed_requires:
+                package, version = parsed_requires[0]
+                requirements.append(f'{package}=={version}')
+            else:
+                print('WARNING! For correct matching dependency links need to specify package name and version'
+                      'such as <dependency url>#egg=<package_name>-<version>')
+        else:
+            requirements.append(requirement)
+    return requirements
+
+
+def get_links():
+    return [
+        requirement for requirement in read('requirements.txt').splitlines()
+        if requirement.startswith('git+') or requirement.startswith('svn+') or requirement.startswith('hg+')
+    ]
+
+
+def get_version():
+    """ Get version from the package without actually importing it. """
+    init = read('deepfloyd_if/__init__.py')
+    for line in init.split('\n'):
+        if line.startswith('__version__'):
+            return eval(line.split('=')[1])
+
+
+setup(
+    name='deepfloyd_if',
+    version=get_version(),
+    author='DeepFloyd, StabilityAI',
+    author_email='[email protected]',
+    description='DeepFloyd-IF (Imagen Free)',
+    packages=['deepfloyd_if', 'deepfloyd_if/model', 'deepfloyd_if/modules', 'deepfloyd_if/pipelines',
+              'deepfloyd_if/resources'],
+    package_data={'deepfloyd_if/resources': ['*.png', '*.npy', '*.npz']},
+    install_requires=get_requirements(),
+    dependency_links=get_links(),
+    long_description=read('README.md'),
+    long_description_content_type='text/markdown',
+)