File size: 3,650 Bytes
9dc6d7f 4362f0a e189fcb 4362f0a 3c12e7b 4362f0a 9dc6d7f 4362f0a 4e5bd0a 4362f0a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 |
# Stable Diffusion 3 Inpainting Pipeline
This is the implementation of `Stable Diffusion 3 Inpainting Pipeline`.
| input image | input mask image | output |
|:-------------------------:|:-------------------------:|:-------------------------:|
|<img src="./overture-creations-5sI6fQgYIuo.png" width = "400" /> | <img src="./overture-creations-5sI6fQgYIuo_mask.png" width = "400" /> | <img src="./overture-creations-5sI6fQgYIuo_output.jpg" width = "400" /> |
|<img src="./overture-creations-5sI6fQgYIuo.png" width = "400" /> | <img src="./overture-creations-5sI6fQgYIuo_mask.png" width = "400" /> | <img src="./overture-creations-5sI6fQgYIuo_tiger.jpg" width = "400" /> |
|<img src="./overture-creations-5sI6fQgYIuo.png" width = "400" /> | <img src="./overture-creations-5sI6fQgYIuo_mask.png" width = "400" /> | <img src="./overture-creations-5sI6fQgYIuo_panda.jpg" width = "400" /> |
**Please ensure that the version of diffusers >= 0.29.1**
## Model
[Stable Diffusion 3 Medium](https://stability.ai/news/stable-diffusion-3-medium) is a Multimodal Diffusion Transformer (MMDiT) text-to-image model that features greatly improved performance in image quality, typography, complex prompt understanding, and resource-efficiency.
For more technical details, please refer to the [Research paper](https://stability.ai/news/stable-diffusion-3-research-paper).
Please note: this model is released under the Stability Non-Commercial Research Community License. For a Creator License or an Enterprise License visit Stability.ai or [contact us](https://stability.ai/license) for commercial licensing details.
### Model Description
- **Developed by:** Stability AI
- **Model type:** MMDiT text-to-image generative model
- **Model Description:** This is a model that can be used to generate images based on text prompts. It is a Multimodal Diffusion Transformer
(https://arxiv.org/abs/2403.03206) that uses three fixed, pretrained text encoders
([OpenCLIP-ViT/G](https://github.com/mlfoundations/open_clip), [CLIP-ViT/L](https://github.com/openai/CLIP/tree/main) and [T5-xxl](https://huggingface.co/google/t5-v1_1-xxl))
## Demo
Make sure you upgrade to the latest version of diffusers: pip install -U diffusers. And then you can run:
```python
import torch
from torchvision import transforms
from pipeline_stable_diffusion_3_inpaint import StableDiffusion3InpaintPipeline
from diffusers.utils import load_image
def preprocess_image(image):
image = image.convert("RGB")
image = transforms.CenterCrop((image.size[1] // 64 * 64, image.size[0] // 64 * 64))(image)
image = transforms.ToTensor()(image)
image = image.unsqueeze(0).to("cuda")
return image
def preprocess_mask(mask):
mask = mask.convert("L")
mask = transforms.CenterCrop((mask.size[1] // 64 * 64, mask.size[0] // 64 * 64))(mask)
mask = transforms.ToTensor()(mask)
mask = mask.to("cuda")
return mask
pipe = StableDiffusion3InpaintPipeline.from_pretrained(
"stabilityai/stable-diffusion-3-medium-diffusers",
torch_dtype=torch.float16,
).to("cuda")
prompt = "Face of a yellow cat, high resolution, sitting on a park bench"
source_image = load_image(
"./overture-creations-5sI6fQgYIuo.png"
)
source = preprocess_image(source_image)
mask = preprocess_mask(
load_image(
"./overture-creations-5sI6fQgYIuo_mask.png"
)
)
image = pipe(
prompt=prompt,
image=source,
mask_image=1-mask,
height=1024,
width=1024,
num_inference_steps=28,
guidance_scale=7.0,
strength=0.6,
).images[0]
image.save("output.png")
```
|