File size: 5,013 Bytes
23740db d21e053 23740db 1c75267 23740db 1c75267 23740db 1c75267 23740db 1c75267 23740db |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 |
---
language:
- en
license: creativeml-openrail-m
library_name: diffusers
tags:
- art
- diffusion
- Interior
---
# KuJiaLe Layout ControlNet
Given the structural elements of the room, such as the walls, floors, and ceilings. Our model auto-completes the furnishing of the room.
The model is trained on [runwayml/stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5) for interior designs.
### Layout ControlNet Example
Keep the room layout consistent, re-furnish the room.
<table style="table-layout: fixed;">
<tr>
<td style="text-align: center; vertical-align: middle; width: 50%">Input</td>
<td style="text-align: center; vertical-align: middle; width: 50%">Output</td>
</td>
<tr>
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/layout_input.jpg" alt="Input" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/layout_output.jpg" alt="Output" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
</tr>
<tr>
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/living_and_dining_room_input.jpg" alt="Input" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/living_and_dining_room_output.jpg" alt="Output" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
</tr>
<tr>
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/living_room_input.png" alt="Input" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/living_room_output.jpg" alt="Output" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
</tr>
<tr>
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/kitchen_input.jpg" alt="Input" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/kitchen_output.jpg" alt="Output" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
</tr>
</tr>
</table>
## News🔥🔥🔥
* June.06, 2024. Our checkpoint Layout-ControlNet are publicly available on [HuggingFace Repo](https://huggingface.co/kujiale-ai/controlnet-layout).
* June.06, 2024. Our Layout-ControlNet demo are publicly available on [HuggingFace Space](https://huggingface.co/spaces/ysmao/Layout-Control).
## Try our HuggingFace demo:
[HuggingFace Space Demo](https://huggingface.co/spaces/ysmao/Layout-Control)
## Checkpoints
* `control_v1_sd15_layout_fp16`: Layout ControlNet checkpoint, for SD15 models.
## Using in 🧨 diffusers
### Layout ControlNet
```python
import torch
from diffusers.utils import load_image
import numpy as np
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline, UniPCMultistepScheduler
controlnet_checkpoint = "kujiale-ai/controlnet-layout"
# Load original image
image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_input.jpg")
depth_image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_depth.jpg").convert("L")
normal_image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_normal.jpg")
segm_image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_segm.jpg")
W, H = image.size
depth_image = depth_image.resize((W, H))
normal_image = normal_image.resize((W, H))
segm_image = segm_image.resize((W, H))
# Prepare Layout Control Image
depth_image = np.array(depth_image, dtype=np.float32) / 255.0
depth_image = torch.from_numpy(depth_image[:, :, None])[None].permute(0, 3, 1, 2)
normal_image = np.array(normal_image, dtype=np.float32)
normal_image = normal_image / 127.5 - 1.0
normal_image = torch.from_numpy(normal_image)[None].permute(0, 3, 1, 2)
segm_image = np.array(segm_image, dtype=np.float32) / 255.0
segm_image = torch.from_numpy(segm_image)[None].permute(0, 3, 1, 2)
control_image = torch.cat([depth_image, normal_image, segm_image], dim=1)
# Initialize pipeline
controlnet = ControlNetModel.from_pretrained(controlnet_checkpoint, subfolder="control_v1_sd15_layout_fp16", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16).to("cuda")
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
image = pipe("A modern bedroom,best quality", num_inference_steps=30, image=control_image, guidance_scale=7).images[0]
image.save('layout_output.jpg')
``` |