File size: 3,534 Bytes
46a65ff 93ad14d d81f975 93ad14d f9d54ce 93ad14d f9d54ce 46a65ff 3cd5592 93ad14d 250fcaf 4555115 3bea1b2 6ba3d40 a5162aa 4fca7f0 3bea1b2 19c50db 49ee526 3cd5592 3bea1b2 3cd5592 f9d54ce 3cd5592 f9d54ce 75dcd42 250fcaf 59c80f8 bfa1fad 93ad14d 1378a3d 250fcaf 93ad14d 250fcaf f9d54ce 069eeaa 3a60af4 93ad14d 3a60af4 93ad14d 3dd5a91 f9d54ce 93ad14d eb7324f 93ad14d eb7324f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
---
license: creativeml-openrail-m
language:
- en
tags:
- text-to-image
- midjourney
- stable-diffusion
- disco-diffusion
- art
- arxiv:2208.12242
inference: true
library_name: diffusers
---
## Paint Journey V2 is [Paint Journey V1](https://huggingface.co/FredZhang7/paint-journey-v1) fine-tuned on 768x768 oil paintings by Midjourney, Open Journey V2, and Disco Diffusion
*Redoing the examples because I recently discovered the endless possibilities with Paint Journey V2.
Paint Journey V2 crafts more stunning masterpieces with more descriptive positive and negative prompts,
although the model can also generate beautiful landscapes with short prompts.*
Begin the prompt with **((oil painting))** to add the oil paint effect. For digital and other painting styles, enter similar prompts as you would for Midjourney (with some tweaks), Stable Diffusion v1.5 (add more styles), Open Journey V2, or Disco Diffusion.
Paint Journey V2's paintings are closer to human-drawn art than Open Journey V2.
Compared to models like Dreamlike Diffusion 1.0, this model tends to generate 768x768 or higher resolution images with reduced noise levels.
This model is also capable of generating stunning portraits at 768x1144 resolution without duplicated faces (with [Camenduru's WebUI](https://github.com/camenduru/stable-diffusion-webui)), a difficult task to models like DreamShaper 3.3.
[](https://colab.research.google.com/github/AMLA-UBC/100-Exploring-the-World-of-Modern-Machine-Learning/blob/main/assets/PaintJourneyV2.ipynb)
## Training
Instead of solely fine-tuning its Unet, Paint Journey V2 focuses on fine-tuning its text encoder with a diverse range of prompts.
This allows for a seamless blend of the digital and oil painting styles into various other types of prompts, resulting in a more natural and dynamic output.
This model was trained on a curated dataset of roughly 300 images hand-picked from Midjourney, [Prompt Hero](https://prompthero.com/), Open Journey V2, and Reddit.
Before training, I used R-ESRGAN 4x on many images to increase their resolution and reduce noise.
To further improve the resolution and reduce noise in generated images, especially when using the model for img2img, use [Paint Journey VAE](./paint_journey_v2.vae.pt) in combination with [Checkpoint](./paint_journey_v2.ckpt).
For example, a Automatic1111's WebUI user can add both files to the `./stable-diffusion-webui/models/Stable-diffusion` folder.
## Examples
*Releasing soon*
## Automatic1111's WebUI
```
git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
```
Download [paint_journey_v2.ckpt](./paint_journey_v2.ckpt) to the `./stable-diffusion-webui/models/Stable-diffusion` folder. Run `webui-user.bat`.
## Diffusers
```bash
pip install --upgrade diffusers
```
```python
from diffusers import StableDiffusionPipeline
import torch
pipe = StableDiffusionPipeline.from_pretrained("FredZhang7/paint-journey-v2")
pipe = pipe.to("cuda")
# Use Prompt Hero for ideas of descriptive (positive) prompts
prompt = "((oil painting)), a boat sailing, night sky, high resolution, uhd, 4 k wallpaper"
image = pipe(prompt).images[0]
image.save("./result.png")
```
## Safety Checker V2
The official [stable diffusion safety checker](https://huggingface.co/CompVis/stable-diffusion-safety-checker) uses up 1.22GB VRAM.
I recommend using [Google Safesearch Mini V2](https://huggingface.co/FredZhang7/google-safesearch-mini-v2) (220MB) to save 1.0GB VRAM. |