Upload 36 files

.gitattributes

@@ -33,3 +33,18 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+img/ctrlnet-depth.gif filter=lfs diff=lfs merge=lfs -text
+img/ctrlnet-ref.gif filter=lfs diff=lfs merge=lfs -text
+img/i2i-e-ddim.gif filter=lfs diff=lfs merge=lfs -text
+img/i2i-e-euler_a.gif filter=lfs diff=lfs merge=lfs -text
+img/i2i-f-ddim-pp.gif filter=lfs diff=lfs merge=lfs -text
+img/i2i-f-ddim.gif filter=lfs diff=lfs merge=lfs -text
+img/i2i-f-euler_a.gif filter=lfs diff=lfs merge=lfs -text
+img/i2i-s-ddim.gif filter=lfs diff=lfs merge=lfs -text
+img/i2i-s-euler_a.gif filter=lfs diff=lfs merge=lfs -text
+img/t2i-e-ddim.gif filter=lfs diff=lfs merge=lfs -text
+img/t2i-e-euler_a.gif filter=lfs diff=lfs merge=lfs -text
+img/t2i-f-ddim.gif filter=lfs diff=lfs merge=lfs -text
+img/t2i-f-euler_a.gif filter=lfs diff=lfs merge=lfs -text
+img/t2i-s-ddim.gif filter=lfs diff=lfs merge=lfs -text
+img/t2i-s-euler_a.gif filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,11 @@
+# meta
+.vscode/
+__pycache__/
+
+# third party tools
+tools/*
+!tools/README.txt
+!tools/*.cmd
+
+# user-wise config files
+postprocess-config.cmd

LICENSE

@@ -0,0 +1,24 @@
+This is free and unencumbered software released into the public domain.
+
+Anyone is free to copy, modify, publish, use, compile, sell, or
+distribute this software, either in source code form or as a compiled
+binary, for any purpose, commercial or non-commercial, and by any
+means.
+
+In jurisdictions that recognize copyright laws, the author or authors
+of this software dedicate any and all copyright interest in the
+software to the public domain. We make this dedication for the benefit
+of the public at large and to the detriment of our heirs and
+successors. We intend this dedication to be an overt act of
+relinquishment in perpetuity of all present and future rights to this
+software under copyright law.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+For more information, please refer to <https://unlicense.org>

README.md

@@ -0,0 +1,293 @@
+# stable-diffusion-webui-prompt-travel
+
+    Travel between prompts in the latent space to make pseudo-animation, extension script for AUTOMATIC1111/stable-diffusion-webui.
+
+----
+
+<p align="left">
+  <a href="https://github.com/Kahsolt/stable-diffusion-webui-prompt-travel/commits"><img alt="Last Commit" src="https://img.shields.io/github/last-commit/Kahsolt/stable-diffusion-webui-prompt-travel"></a>
+  <a href="https://github.com/Kahsolt/stable-diffusion-webui-prompt-travel/issues"><img alt="GitHub issues" src="https://img.shields.io/github/issues/Kahsolt/stable-diffusion-webui-prompt-travel"></a>
+  <a href="https://github.com/Kahsolt/stable-diffusion-webui-prompt-travel/stargazers"><img alt="GitHub stars" src="https://img.shields.io/github/stars/Kahsolt/stable-diffusion-webui-prompt-travel"></a>
+  <a href="https://github.com/Kahsolt/stable-diffusion-webui-prompt-travel/network"><img alt="GitHub forks" src="https://img.shields.io/github/forks/Kahsolt/stable-diffusion-webui-prompt-travel"></a>
+  <img alt="Language" src="https://img.shields.io/github/languages/top/Kahsolt/stable-diffusion-webui-prompt-travel">
+  <img alt="License" src="https://img.shields.io/github/license/Kahsolt/stable-diffusion-webui-prompt-travel">
+  <br/>
+</p>
+
+![:stable-diffusion-webui-prompt-travel](https://count.getloli.com/get/@:stable-diffusion-webui-prompt-travel)
+
+Try interpolating on the hidden vectors of conditioning prompt to make seemingly-continuous image sequence, or let's say a pseudo-animation. 😀  
+Not only prompts! We also support non-prompt conditions, read => [README_ext.md](README_ext.md) ~  
+
+⚠ 我们成立了插件反馈 QQ 群: 616795645 (赤狐屿)，欢迎出建议、意见、报告bug等 (w  
+⚠ We have a QQ chat group (616795645) now, any suggestions, discussions and bug reports are highly wellllcome!!  
+
+ℹ 实话不说，我想有可能通过这个来做ppt童话绘本<del>甚至本子</del>……  
+ℹ 聪明的用法：先手工盲搜两张好看的图 (只有prompt差异)，然后再尝试在其间 travel :lolipop:  
+
+⚠ Remeber to check "Always save all generated images" on in the settings tab, otherwise "upscaling" and "saving intermediate images" would not work.
+⚠ 记得在设置页勾选 “总是保存所有生成的图片”，否则 上采样 与 保存中间图片 将无法工作。
+
+
+### Change Log
+
+⚪ Compatibility
+
+The latest version `v3.0` is synced & tested with:
+
+- [AUTOMATIC1111/stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui): version `v1.4.0`, tag [v1.4.0](https://github.com/AUTOMATIC1111/stable-diffusion-webui/releases/tag/v1.4.0)
+- [Mikubill/sd-webui-controlnet](https://github.com/Mikubill/sd-webui-controlnet): version `v1.1.229`, commit [eceeec7a7e](https://github.com/Mikubill/sd-webui-controlnet/commit/eceeec7a7e856867de56e26cae9f3e1076480344)
+
+⚪ Features
+
+- 2023/07/05: `v3.0` re-impl core with sd-webui `v1.4.0` callbacks; this new implementation will be slower, but more compatible with other extensions
+- 2023/04/13: `v2.7` add RIFE to controlnet-travel, skip fusion (experimental)
+- 2023/03/31: `v2.6` add a tkinter [GUI](#run-each-time) for postprocess toolchain
+- 2023/03/30: `v2.5` add controlnet-travel script (experimental), interpolating between hint conditions **instead of prompts**, thx for the code base from [sd-webui-controlnet](https://github.com/Mikubill/sd-webui-controlnet) 
+- 2023/02/14: `v2.3` integrate basic function of [depth-image-io](https://github.com/AnonymousCervine/depth-image-io-for-SDWebui) for depth2img models
+- 2023/01/27: `v2.2` add 'slerp' linear interpolation method
+- 2023/01/22: `v2.1` add experimental 'replace' mode again, it's not smooth interpolation
+- 2023/01/20: `v2.0` add optional external [post-processing pipeline](#post-processing-pipeline) to highly boost up smoothness, greate thx to [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN) and [RIFE](https://github.com/nihui/rife-ncnn-vulkan)!!
+- 2023/01/16: `v1.5` add upscale options (issue #12); add 'embryo' genesis, reproducing idea of [stable-diffusion-animation](https://replicate.com/andreasjansson/stable-diffusion-animation) except [FILM](https://github.com/google-research/frame-interpolation) support (issue #11)
+- 2023/01/12: `v1.4` remove 'replace' & 'grad' mode support, due to webui's code change
+- 2022/12/11: `v1.3` work in a more 'successive' way, idea borrowed from [deforum](https://github.com/deforum-art/deforum-for-automatic1111-webui) ('genesis' option)
+- 2022/11/14: `v1.2` walk by substituting token embedding ('replace' mode)
+- 2022/11/13: `v1.1` walk by optimizing condition ('grad' mode)
+- 2022/11/10: `v1.0` interpolate linearly on condition/uncondition ('linear' mode)
+
+⚪ Fixups
+
+- 2023/07/05: sync sd-webui-controlnet to `v1.1.229`
+- 2023/04/30: update controlnet core to `v1.1.116`
+- 2023/03/29: `v2.4` bug fixes on script hook, now working correctly with extra networks & [sd-webui-controlnet](https://github.com/Mikubill/sd-webui-controlnet)
+- 2023/01/31: keep up with webui's updates, (issue #14: `ImportError: cannot import name 'single_sample_to_image'`)
+- 2023/01/28: keep up with webui's updates, extra-networks rework
+- 2023/01/16: `v1.5` apply zero padding when condition length mismatch (issue #10: `RuntimeError: The size of tensor a (77) must match the size of tensor b (154) at non-singleton dimension 0`), typo in demo filename
+- 2023/01/12: `v1.4` keep up with webui's updates (issue #9: `AttributeError: 'FrozenCLIPEmbedderWithCustomWords' object has no attribute 'process_text'`)
+- 2022/12/13: `#bdd8bed` fixup no working when negative prompt is left empty (issue #6: `neg_prompts[-1] IndexError: List index out of range`)
+- 2022/11/27: `v1.2-fix2` keep up with webui's updates (error `ImportError: FrozenCLIPEmbedderWithCustomWords`)
+- 2022/11/20: `v1.2-fix1` keep up with webui's updates (error `AttributeError: p.all_negative_prompts[0]`)
+
+⚠ this script will NOT probably support the schedule syntax (i.e.: `[prompt:prompt:number]`), because interpolate on mutable conditions requires sampler level tracing which is hard to maintain :(  
+⚠ this script will NOT probably work together with `hires.fix` due to some inner conceptual/logical conflict of `denoising_strength`, you can alternatively perform batch-upscale then batch-img2img.  
+
+
+### How it works?
+
+- input **multiple lines** in the prompt/negative-prompt box, each line is called a **stage**
+- generate images one by one, interpolating from one stage towards the next (batch configs are ignored)
+- gradually change the digested inputs between prompts
+  - freeze all other settings (`steps`, `sampler`, `cfg factor`, `seed`, etc.)
+  - note that only the major `seed` will be forcely fixed through all processes, you can still set `subseed = -1` to allow more variances
+- export a video!
+  - follow [post-processing pipeline](#post-processing-pipeline) to get much better result 👌
+
+⚪ Txt2Img
+
+| sampler \ genesis | fixed | successive | embryo |
+| :-: | :-: | :-: | :-: |
+| Eular a | ![t2i-f-euler_a](img/t2i-f-euler_a.gif) | ![t2i-s-euler_a](img/t2i-s-euler_a.gif) | ![t2i-e-euler_a](img/t2i-e-euler_a.gif) |
+| DDIM    | ![t2i-f-ddim](img/t2i-f-ddim.gif)       | ![t2i-s-ddim](img/t2i-s-ddim.gif)       | ![t2i-e-ddim](img/t2i-e-ddim.gif)       |
+
+⚪ Img2Img
+
+| sampler \ genesis | fixed | successive | embryo |
+| :-: | :-: | :-: | :-: |
+| Eular a | ![i2i-f-euler_a](img/i2i-f-euler_a.gif) | ![i2i-s-euler_a](img/i2i-s-euler_a.gif) | ![i2i-e-euler_a](img/i2i-e-euler_a.gif) |
+| DDIM    | ![i2i-f-ddim](img/i2i-f-ddim.gif)       | ![i2i-s-ddim](img/i2i-s-ddim.gif)       | ![i2i-e-ddim](img/i2i-e-ddim.gif)       |
+
+post-processing pipeline (case `i2i-f-ddim`):
+
+| w/o. post-processing | w/. post-processing |
+| :-: | :-: |
+| ![i2i-f-ddim](img/i2i-f-ddim.gif) | ![i2i-f-ddim-pp](img/i2i-f-ddim-pp.gif) |
+
+other stuff:
+
+| reference image for img2img | embryo image decoded <br/> case `i2i-e-euler_a` with `embryo_step=8` |
+| :-: | :-: |
+| ![i2i-ref](img/i2i-ref.png) | ![embryo](img/embryo.png) |
+
+⚪ ControlNet support
+
+| prompt-travel with ControlNet (depth) | controlnet-travel (depth) |
+| :-: | :-: |
+| ![ctrlnet-ref](img/ctrlnet-ref.gif) | ![ctrlnet-depth](img/ctrlnet-depth.gif) |
+
+
+Example above run configure:
+
+```text
+Prompt:
+(((masterpiece))), highres, ((boy)), child, cat ears, white hair, red eyes, yellow bell, red cloak, barefoot, angel, [flying], egyptian
+((masterpiece)), highres, ((girl)), loli, cat ears, light blue hair, red eyes, magical wand, barefoot, [running]
+
+Negative prompt:
+(((nsfw))), ugly,duplicate,morbid,mutilated,tranny,trans,trannsexual,mutation,deformed,long neck,bad anatomy,bad proportions,extra arms,extra legs, disfigured,more than 2 nipples,malformed,mutated,hermaphrodite,out of frame,extra limbs,missing arms,missing legs,poorly drawn hands,poorty drawn face,mutation,poorly drawn,long body,multiple breasts,cloned face,gross proportions, mutated hands,bad hands,bad feet,long neck,missing limb,malformed limbs,malformed hands,fused fingers,too many fingers,extra fingers,missing fingers,extra digit,fewer digits,mutated hands and fingers,lowres,text,error,cropped,worst quality,low quality,normal quality,jpeg artifacts,signature,watermark,username,blurry,text font ufemale focus, poorly drawn, deformed, poorly drawn face, (extra leg:1.3), (extra fingers:1.2),out of frame
+
+Steps: 15
+CFG scale: 7
+Clip skip: 1
+Seed: 114514
+Size: 512 x 512
+Model hash: animefull-final-pruned.ckpt
+Hypernet: (this is my secret :)
+```
+
+
+### Options
+
+- prompt: (list of strings)
+- negative prompt: (list of strings)
+  - input multiple lines of prompt text
+  - we call each line of prompt a stage, usually you need at least 2 lines of text to starts travel
+  - if len(positive_prompts) != len(negative_prompts), the shorter one's last item will be repeated to match the longer one
+- mode: (categorical)
+  - `linear`: linear interpolation on condition/uncondition of CLIP output
+  - `replace`: gradually replace of CLIP output
+    - replace_dim: (categorical)
+      - `token`: per token-wise vector
+      - `channel`: per channel-wise vector
+      - `random`: per point-wise element
+    - replace_order: (categorical)
+      - `similiar`: from the most similiar first (L1 distance)
+      - `different`: from the most different first
+      - `random`: just randomly
+  - `embryo`: pre-denoise few steps, then hatch a set of image from the common embryo by linear interpolation
+- steps: (int, list of int)
+  - number of images to interpolate between two stages
+  - if int, constant number of travel steps
+  - if list of int, length should match `len(stages)-1`, separate by comma, e.g.: `12, 24, 36`
+- genesis: (categorical), the a prior for each image frame
+  - `fixed`: starts from pure noise in txt2img pipeline, or from the same ref-image given in img2img pipeline
+  - `successive`: starts from the last generated image (this will force txt2img turn to actually be img2img from the 2nd frame on)
+  - `embryo`: starts from the same half-denoised image, see [=> How does it work?](https://replicate.com/andreasjansson/stable-diffusion-animation#readme) 
+    - (experimental) it only processes 2 lines of prompts, and does not interpolate on negative_prompt :(
+- genesis_extra_params
+  - denoise_strength: (float), denoise strength in img2img pipelines (for `successive`)
+  - embryo_step: (int or float), steps to hatch the common embryo (for `embryo`)
+    - if >= 1, taken as step cout
+    - if < 1, taken as ratio of total step
+- video_*
+  - fps: (float), FPS of video, set `0` to disable file saving
+  - fmt: (categorical), export video file format
+  - pad: (int), repeat beginning/ending frames, giving a in/out time
+  - pick: (string), cherry pick frames by [python slice syntax](https://www.pythoncentral.io/how-to-slice-listsarrays-and-tuples-in-python) before padding (e.g.: set `::2` to get only even frames, set `:-1` to drop last frame)
+
+
+### Installation
+
+Easiest way to install it is to:
+1. Go to the "Extensions" tab in the webui, switch to the "Install from URL" tab
+2. Paste https://github.com/Kahsolt/stable-diffusion-webui-prompt-travel.git into "URL for extension's git repository" and click install
+3. (Optional) You will need to restart the webui for dependencies to be installed or you won't be able to generate video files
+
+Manual install:
+1. Copy this repo folder to the 'extensions' folder of https://github.com/AUTOMATIC1111/stable-diffusion-webui
+2. (Optional) Restart the webui
+
+
+### Post-processing pipeline
+
+There are still two steps away from a really smooth and high resolution animation, namely image **super-resolution** & video **frame interpolation** (see `third-party tools` below).  
+⚠ Media data processing is intrinsic resource-exhausting, and it's also not webui's work or duty, hence we separated it out. 😃
+
+#### setup once
+
+⚪ auto install (Windows)
+
+- run `cd tools & install.cmd`
+- trouble shooting
+  - if you got any file system access errors like `Access denied.`, try run it again until you see `Done!` without errors 😂
+  - if you got SSL errors about `curl schannel ... Unknown error ... certificate.`, the downloader not work due to some SSL security reasons, just turn to install manually...
+- you will have four components: [Busybox](https://frippery.org/busybox/), [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan), [RIFE](https://github.com/nihui/rife-ncnn-vulkan) and [FFmpeg](https://ffmpeg.org/) installed under the [tools](tools) folder
+
+⚪ manually install (Windows/Linux/Mac)
+
+ℹ Understand the `tools` folder layout first => [tools/README.txt](tools/README.txt)  
+ℹ If you indeed wanna put the tools elsewhere, modify paths in [tools/link.cmd](tools/link.cmd) and run `cd tools & link.cmd` 😉
+
+For Windows: 
+
+- download [Busybox](https://frippery.org/busybox/)
+- download [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN/releases) (e.g.: `realesrgan-ncnn-vulkan-20220424-windows.zip`)
+  - (optional) download interesting seperated model checkpoints (e.g.: `realesr-animevideov3.pth`)
+- download [rife-ncnn-vulkan](https://github.com/nihui/rife-ncnn-vulkan/releases) bundle (e.g.: `rife-ncnn-vulkan-20221029-windows.zip `)
+- download [FFmpeg](https://ffmpeg.org/download.html) binary (e.g.: `ffmpeg-release-full-shared.7z` or `ffmpeg-git-full.7z`)
+
+For Linux/Mac: 
+
+- download [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN/releases) and [rife-ncnn-vulkan](https://github.com/nihui/rife-ncnn-vulkan/releases), put them according to the `tools` folder layout, manually apply `chmod 755` to the executables
+- `ffmpeg` can be easily found in your app store or package manager, run like `apt install ffmpeg`; DO NOT need to link it under `tools` folder
+
+
+#### run each time
+
+⚪ tkinter GUI (Windows/Linux/Mac)
+
+![manager](img/manager.png)
+
+For Windows:
+  - run `manager.cmd`, to start webui's python venv
+  - run the [DOSKEY](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/doskey) `install` (only setup once)
+  - run the DOSKEY `run`
+
+For Linux/Mac:
+  - run `../../venv/Scripts/activate`, to start webui's python venv
+  - run `pip install -r requirements.txt` (only setup once)
+  - run `python manager.py`
+
+ℹ find usage help message in right click pop menu~
+
+⚪ <del> cmd script (Windows) - deprecated </del>
+
+- check params in [postprocess-config.cmd](postprocess-config.cmd)
+- pick one way to start 😃
+  - run `postprocess.cmd path/to/<image_folder>` from command line
+  - drag & drop any image folder over `postprocess.cmd` icon
+- once processing finished, the explorer will be auto lauched to locate the generated file named with `synth.mp4`
+
+
+### Related Projects
+
+⚪ extensions that inspired this repo
+
+- sd-webui-controlnet (various image conditions): [https://github.com/Mikubill/sd-webui-controlnet](https://github.com/Mikubill/sd-webui-controlnet)
+- depth-image-io (custom depth2img): [https://github.com/AnonymousCervine/depth-image-io-for-SDWebui](https://github.com/AnonymousCervine/depth-image-io-for-SDWebui)
+- animator (img2img): [https://github.com/Animator-Anon/animator_extension](https://github.com/Animator-Anon/animator_extension)
+- sd-webui-riffusion (music gen): [https://github.com/enlyth/sd-webui-riffusion](https://github.com/enlyth/sd-webui-riffusion)
+- sd-animation (half denoise + FILM):
+  - Github: [https://github.com/andreasjansson/cog-stable-diffusion](https://github.com/andreasjansson/cog-stable-diffusion)
+  - Replicate: [https://replicate.com/andreasjansson/stable-diffusion-animation](https://replicate.com/andreasjansson/stable-diffusion-animation)
+- deforum (img2img + depth model): [https://github.com/deforum-art/deforum-for-automatic1111-webui](https://github.com/deforum-art/deforum-for-automatic1111-webui)
+- seed-travel (varying seed): [https://github.com/yownas/seed_travel](https://github.com/yownas/seed_travel)
+
+⚪ third-party tools
+
+- image super-resoultion
+  - ESRGAN:
+    - ESRGAN: [https://github.com/xinntao/ESRGAN](https://github.com/xinntao/ESRGAN)
+    - Real-ESRGAN: [https://github.com/xinntao/Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN)
+    - Real-ESRGAN-ncnn-vulkan (recommended): [https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan)
+- video frame interpolation
+  - FILM (recommended): [https://github.com/google-research/frame-interpolation](https://github.com/google-research/frame-interpolation)
+  - RIFE:
+    - ECCV2022-RIFE: [https://github.com/megvii-research/ECCV2022-RIFE](https://github.com/megvii-research/ECCV2022-RIFE)
+    - rife-ncnn-vulkan (recommended): [https://github.com/nihui/rife-ncnn-vulkan](https://github.com/nihui/rife-ncnn-vulkan)
+    - Squirrel-RIFE: [https://github.com/Justin62628/Squirrel-RIFE](https://github.com/Justin62628/Squirrel-RIFE)
+    - Practical-RIFE: [https://github.com/hzwer/Practical-RIFE](https://github.com/hzwer/Practical-RIFE)
+- GNU tool-kits
+  - BusyBox: [https://www.busybox.net/](https://www.busybox.net/)
+    - BusyBox for Windows: [https://frippery.org/busybox/](https://frippery.org/busybox/)
+  - FFmpeg: [https://ffmpeg.org/](https://ffmpeg.org/)
+
+⚪ my other experimental toy extensions
+
+- vid2vid (video2video) [https://github.com/Kahsolt/stable-diffusion-webui-vid2vid](https://github.com/Kahsolt/stable-diffusion-webui-vid2vid)
+- hires-fix-progressive (a progressive version of hires.fix): [https://github.com/Kahsolt/stable-diffusion-webui-hires-fix-progressive](https://github.com/Kahsolt/stable-diffusion-webui-hires-fix-progressive)
+- sonar (k_diffuison samplers): [https://github.com/Kahsolt/stable-diffusion-webui-sonar](https://github.com/Kahsolt/stable-diffusion-webui-sonar)
+- size-travel (kind of X-Y plot on image size): [https://github.com/Kahsolt/stable-diffusion-webui-size-travel](https://github.com/Kahsolt/stable-diffusion-webui-size-travel)
+
+----
+by Armit
+2022/11/10 

README.zh_CN.md

@@ -0,0 +1,91 @@
+# 提示词跃迁
+
+    在模型隐层旅行以制作伪动画，项目 AUTOMATIC1111/stable-diffusion-webui 的插件。
+
+----
+
+对语言理解模型 CLIP 的输出进行插值，从而实现多条提示词之间的语义过渡，产生看似连续的图像序列，或者说伪动画。😀  
+
+⚠ 我们成立了插件反馈 QQ 群: 616795645 (赤狐屿)，欢迎出建议、意见、报告bug等 (w  
+
+ℹ 实话不说，我想有可能通过这个来做ppt童话绘本<del>甚至本子</del>……  
+ℹ 聪明的用法：先手工盲搜两张好看的图 (只有提示词差异)，然后再尝试在其间跃迁 :lolipop:  
+
+
+### 使用方法 & 它如何工作
+
+- 在提示词/负向提示词框里输入**多行**文本，每一行被称作一个**阶段**
+- 逐帧生成图像，在每个阶段内，所使用的提示词向量是经过插值运算的
+  - 为了保证某种连续性，所有其他参数将被固定
+  - 虽然所有图的主随机数种子将被统一固定，但你仍然可以启用 `subseed` 去增加随机性
+- 导出视频！
+  - 使用额外的 [后处理流程](#post-processing-pipeline) 可以获得更好的画质和流畅度 👌
+
+
+### 参数选项
+
+- 提示词: (多行文本)
+- 反向提示词: (多行文本)
+  - 就是提示词和反向提示词的输入框，但是你必须输入多行文本，每一行是一个阶段
+  - 如果提示词和反向提示词的阶段数量不一致，少的那一方会被重复到对齐多的一方
+- 插帧数/steps: (整数，或者逗号分隔的多个整数)
+  - 每个阶段之间插帧的数量
+  - 若为单个整数，每个阶段使用相同的插帧数量
+  - 若为西文逗号分隔的多个整数，每个阶段使用不同的插帧数量，比如有4个阶段则可给出3个独立步数：`12, 24, 36`
+- 起源/genesis: (选项), 每张图像的内容先验
+  - `固定/fixed`: 在 txt2img 流程中，始终从高斯噪声开始降噪；在 img2img 流程中，始终从给定的参考图开始降噪
+  - `连续/successive`: 从上一帧的内容开始降噪 (这会导致 txt2img 流程从第二步开始强制转为 img2img 流程)
+  - `胚胎/embryo`: 从某个已部分降噪的公共先祖胚胎开始降噪，参考 [=> 原理](https://replicate.com/andreasjansson/stable-diffusion-animation#readme) 
+    - (该功能为实验性质) 只支持两个阶段跃迁，并且不能为逆向提示词插值 :(
+- 起源的额外参数
+  - 降噪强度: (浮点数), 在 img2img 流程中所用的降噪强度 (仅对 `连续/successive` 模式)
+  - 胚胎步数: (整数或浮点数), 产生公共胚胎的预降噪步数 (仅对 `胚胎/embryo` 模式)
+    - 如果 >= 1，解释为采样步数
+    - 如果 < 1，解释为占总采样步数的比例
+- 视频相关
+  - 帧率/fps: (浮点数), 导出视频的帧率，设置为 `0` 将禁用导出
+  - 文件格式/fmt: (选项), 导出视频的文件格式
+  - 首尾填充/pad: (整数), 重复首尾帧 `N` 次以留出一段入场/退场时间
+  - 帧选择器/pick: (切片器), 使用 [Python切片语法](https://www.pythoncentral.io/how-to-slice-listsarrays-and-tuples-in-python) 精心选择所需要导出的帧，注意切片发生在填充之前 (例如：设为 `::2` 将只使用偶数帧 , 设为 `:-1` 将去除最后一帧)
+- 调试开关: (逻辑值)
+  - 是否在控制台显示详细日志
+
+
+### 后处理流程
+
+单凭CLIP模型自身能实现语义插值就已经到达能力天花板了，但我们距离高清丝滑的动画还差两步： **图像超分辨率** 和 **视频插帧**。 
+⚠ 多媒体数据的处理是非常消耗资源的，我们不能指望 webui 去做这件事。实际上，我们将其从宿主和插件中分离，出来作为一个可选的外部工具。 😃
+
+#### 安装依赖
+
+⚪ 自动安装
+
+- 运行 `tools/install.cmd`
+- 如果遇到诸如 `访问被拒绝` 之类的错误，多次运行直到提示 `Done!` 无错误退出 😂
+- 你将安装好 [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan), [RIFE](https://github.com/nihui/rife-ncnn-vulkan), [FFmpeg](https://ffmpeg.org/) 这三个组件在 [tools](tools) 目录下
+
+⚪ 手动安装
+
+- 参照 [README.md](README.md#post-processing-pipeline)
+- 我寻思你既然都想着手动安装了，也不至于不肯咬一口英文罢…… 🤔
+
+#### 运行任务
+
+- 检查 [postprocess.cmd](postprocess.cmd) 中的默认参数
+- 你有两种方式启动后处理任务 😃
+  - 从命令行运行 `postprocess.cmd path/to/<image_folder>`
+  - 鼠标拖拽任意图片文件夹到 `postprocess.cmd` 的文件图标上然后释放
+
+ℹ 任务完成后，资源浏览器将被自动打开并定位到导出的 `synth.mp4` 文件~
+
+
+插件直出和加入后处理对比 (配置为 `img2img-fixed-ddim`)：
+
+| 插件直出 | 加入后处理 |
+| :-: | :-: |
+| ![i2i-f-ddim](img/i2i-f-ddim.gif) | ![i2i-f-ddim-pp](img/i2i-f-ddim-pp.gif) |
+
+
+----
+by Armit
+2023/01/20 

README_ext.md

@@ -0,0 +1,34 @@
+# stable-diffusion-webui-non-prompt-travel (extensions)
+
+    Of course not only prompts! -- You shall also be able to travel through any other conditions. 😀
+
+----
+
+### ControlNet-Travel
+
+Travel through ControlNet's control conditions like canny, depth, openpose, etc...
+
+⚠ Memory (not VRAM) usage grows linearly with sampling steps, and fusion layers count, this is its nature 😥
+
+Quickstart instructions:
+
+- prepare a folder of images, might be frames from a video
+- check enble `sd-webui-controlnet`, set all parameters as you want, but it's ok to **leave the ref image box empty**
+  - reference images will be read from the image folder given in controlnet-travel :)
+- find `ControlNet Travel` in the script dropdown, set all parameters again, specify your image folder path here
+- click Generate button
+
+Options:
+
+- interp_meth: (categorical)
+  - `linear`: linear weighted sum, better for area-based annotaions like `depth`, `seg`
+  - `rife`: optical flow model (requires to install postprocess tools first), better for edge-base annotaions like `canny`, `openpose`
+- skip_latent_fusion: (list of bool), experimental
+  - skip some latent layers fusion for saving memory, but might get wierd results 🤔
+  - ℹ in my experiences, the `mid` and `in` blocks are more safe to skip
+- save_rife: (bool), save the rife interpolated condtion images
+
+
+----
+by Armit
+2023/04/12

install.py

@@ -0,0 +1,4 @@
+import launch
+
+if not launch.is_installed("moviepy"):
+    launch.run_pip("install moviepy==1.0.3", "requirements for Prompt Travel to generate video")

manager.cmd

@@ -0,0 +1,41 @@
+@REM start webui's python venv
+@ECHO OFF
+
+SET SD_PATH=%~dp0\..\..
+PUSHD %SD_PATH%
+SET SD_PATH=%CD%
+POPD
+
+REM SET VENV_PATH=C:\Miniconda3
+SET VENV_PATH=%SD_PATH%\venv
+
+SET PATH=%VENV_PATH%\Scripts;%PATH%
+SET PY_BIN=python.exe
+
+%PY_BIN% --version > NUL
+IF ERRORLEVEL 1 GOTO die
+
+DOSKEY run=python manager.py
+DOSKEY install=pip install -r requirements.txt
+
+CMD /K activate.bat ^& ^
+  ECHO VENV_PATH: %VENV_PATH% ^& ^
+  %PY_BIN% --version ^& ^
+  ECHO. ^& ^
+  ECHO Commands shortcuts: ^& ^
+  ECHO   run        start ptravel manager ^& ^
+  ECHO   install    install requirements.txt
+
+
+GOTO EOF
+
+:die
+ECHO ERRORLEVEL: %ERRORLEVEL%
+ECHO PATH: %PATH%
+ECHO VENV_PATH: %VENV_PATH%
+ECHO Python executables:
+WHERE python.exe
+
+PAUSE
+
+:EOF

manager.py

@@ -0,0 +1,513 @@
+#!/usr/bin/env pythonw3
+# Author: Armit
+# Create Time: 2023/03/31 
+
+import sys
+import os
+import shutil
+import psutil
+from pathlib import Path
+from time import time
+from PIL import Image
+from PIL.ImageTk import PhotoImage
+import subprocess
+from subprocess import Popen
+from threading import Thread
+from typing import Union
+import gc
+
+import tkinter as tk
+import tkinter.ttk as ttk
+import tkinter.messagebox as tkmsg
+import tkinter.filedialog as tkfdlg
+from traceback import print_exc, format_exc
+
+__version__ = '0.1'
+
+BASE_PATH = Path(__file__).absolute().parent
+WEBUI_PATH = BASE_PATH.parent.parent
+OUTPUT_PATH = WEBUI_PATH / 'outputs'
+DEFAULT_OUTPUT_PATH = OUTPUT_PATH / 'txt2img-images' / 'prompt_travel'
+
+TOOL_PATH = BASE_PATH / 'tools'
+paths_ext = []
+paths_ext.append(str(TOOL_PATH))
+paths_ext.append(str(TOOL_PATH / 'realesrgan-ncnn-vulkan'))
+paths_ext.append(str(TOOL_PATH / 'rife-ncnn-vulkan'))
+paths_ext.append(str(TOOL_PATH / 'ffmpeg'))
+os.environ['PATH'] += os.path.pathsep + os.path.pathsep.join(paths_ext)
+
+RESR_MODELS = {
+  'realesr-animevideov3': [2, 3, 4],
+  'realesrgan-x4plus-anime': [4],
+  'realesrgan-x4plus': [4],
+}
+RIFE_MODELS = [
+  'rife',
+  'rife-anime',
+  'rife-HD',
+  'rife-UHD',
+  'rife-v2',
+  'rife-v2.3',
+  'rife-v2.4',
+  'rife-v3.0',
+  'rife-v3.1',
+  'rife-v4',
+  'rife-v4.6',
+]
+EXPORT_FMT  = [
+  'mp4',
+  'gif',
+  'webm',
+]
+
+def sanitize_pathname(path: Union[str, Path]) -> str:
+  if isinstance(path, Path): path = str(path)
+  return path.replace('\\', os.path.sep)
+
+def startfile(path:Union[str, Path]):
+  # ref: https://stackoverflow.com/questions/17317219/is-there-an-platform-independent-equivalent-of-os-startfile/17317468#17317468
+  if isinstance(path, Path): path = str(path)
+  if sys.platform == 'win32':
+    os.startfile(path)
+  else:
+    opener = "open" if sys.platform == "darwin" else "xdg-open"
+    subprocess.call([opener, path])
+
+def run_cmd(cmd:str) -> bool:
+  try:
+    print(f'[exec] {cmd}')
+    Popen(cmd, shell=True, encoding='utf-8').wait()
+    return True
+  except:
+    return False
+
+def run_resr(model:str, ratio:int, in_dp:Path, out_dp:Path) -> bool:
+  if out_dp.exists(): shutil.rmtree(str(out_dp))
+  out_dp.mkdir(exist_ok=True)
+
+  if model == 'realesr-animevideov3': model = f'realesr-animevideov3-x{ratio}'
+  safe_out_dp = sanitize_pathname(out_dp)
+  ok = run_cmd(f'realesrgan-ncnn-vulkan -v -s {ratio} -n {model} -i "{sanitize_pathname(in_dp)}" -o "{safe_out_dp}"')
+
+  # NOTE: fix case of Embryo mode
+  embryo_fp: Path = out_dp / 'embryo.png'
+  if embryo_fp.exists(): embryo_fp.unlink()
+
+  return ok
+
+def run_rife(model:str, interp:int, in_dp:Path, out_dp:Path) -> bool:
+  if out_dp.exists(): shutil.rmtree(str(out_dp))
+  out_dp.mkdir(exist_ok=True)
+
+  if model == 'rife-v4':
+    if interp > 0: interp *= len(list(in_dp.iterdir()))
+    return run_cmd(f'rife-ncnn-vulkan -v -n {interp} -m {model} -i "{sanitize_pathname(in_dp)}" -o "{sanitize_pathname(out_dp)}"')
+  else:
+    return run_cmd(f'rife-ncnn-vulkan -v -m {model} -i "{sanitize_pathname(in_dp)}" -o "{sanitize_pathname(out_dp)}"')
+
+def run_ffmpeg(fps:float, fmt:str, in_dp:Path, out_dp:Path) -> bool:
+  out_fp = out_dp / f'synth.{fmt}'
+  if out_fp.exists(): out_fp.unlink()
+
+  if fmt == 'gif':
+    return run_cmd(f'ffmpeg -y -framerate {fps} -i "{sanitize_pathname(in_dp / r"%08d.png")}" "{sanitize_pathname(out_fp)}"')
+  if fmt == 'mp4':
+    return run_cmd(f'ffmpeg -y -framerate {fps} -i "{sanitize_pathname(in_dp / r"%08d.png")}" -crf 30 -c:v libx264 -pix_fmt yuv420p "{sanitize_pathname(out_fp)}"')
+  if fmt == 'webm':
+    # -c:v libvpx/libvpx-vp9/libaom-av1 (VP8/VP9/AV1)
+    # -b:v 0/1M
+    # -crf 15~30
+    return run_cmd(f'ffmpeg -y -framerate {fps} -i "{sanitize_pathname(in_dp / r"%08d.png")}" -crf 30 -c:v libvpx-vp9 -pix_fmt yuv420p "{sanitize_pathname(out_fp)}"')
+
+
+WINDOW_TITLE  = f'Prompt Travel Manager v{__version__}'
+WINDOW_SIZE   = (710, 660)
+IMAGE_SIZE    = 512
+LIST_HEIGHT   = 100
+COMBOX_WIDTH  = 18
+COMBOX_WIDTH1 = 4
+ENTRY_WIDTH   = 7
+MEMINFO_REFRESH = 16    # refresh status memory info every k-image loads
+
+HELP_INFO = '''
+[Settings]
+  resr: model_name, upscale_ratio
+    - only realesr-animevideov3 supports custom upscale_ratio
+    - others are forced x4
+  rife: model_name, interp_ratio (NOT frame count!!)
+    - only rife-v4 supports custom interp_ratio
+    - others are forced x2
+  ffmpeg: export_format, export_fps
+
+The checkboxes are enable switches specifying to run or not :)
+'''
+
+
+class App:
+
+  def __init__(self):
+    self.setup_gui()
+
+    self.is_running = False
+    self.cur_name = None  # str, current travel id
+    self.cache = {}       # { 'name': [Image|Path] }
+
+    self.p = psutil.Process(os.getpid())
+    self.cnt_pv_load = 0
+
+    if DEFAULT_OUTPUT_PATH.exists():
+      self.open_(DEFAULT_OUTPUT_PATH)
+    self.var_status.set(self._mem_info_str())
+
+    try:
+      self.wnd.mainloop()
+    except KeyboardInterrupt:
+      self.wnd.quit()
+    except: print_exc()
+
+  def setup_gui(self):
+    # window
+    wnd = tk.Tk()
+    W, H = wnd.winfo_screenwidth(), wnd.winfo_screenheight()
+    w, h = WINDOW_SIZE
+    wnd.geometry(f'{w}x{h}+{(W-w)//2}+{(H-h)//2}')
+    wnd.resizable(False, False)
+    wnd.title(WINDOW_TITLE)
+    wnd.protocol('WM_DELETE_WINDOW', wnd.quit)
+    self.wnd = wnd
+
+    # menu
+    menu = tk.Menu(wnd, tearoff=0)
+    menu.add_command(label='Open folder...', command=self._menu_open_dir)
+    menu.add_command(label='Remove folder', command=self._menu_remove_dir)
+    menu.add_separator()
+    menu.add_command(label='Memory cache clean', command=self.mem_clear)
+    menu.add_command(label='Help', command=lambda: tkmsg.showinfo('Help', HELP_INFO))
+    def menu_show(evt):
+      try:     menu.tk_popup(evt.x_root, evt.y_root)
+      finally: menu.grab_release()
+
+    # top: travel folder
+    frm1 = ttk.LabelFrame(wnd, text='Travel root folder')
+    frm1.pack(side=tk.TOP, anchor=tk.N, expand=tk.YES, fill=tk.X)
+    if True:
+      self.var_root_dp = tk.StringVar(wnd)
+      tk.Entry(frm1, textvariable=self.var_root_dp).pack(side=tk.LEFT, expand=tk.YES, fill=tk.X)
+      tk.Button(frm1, text='Open..', command=self.open_).pack(side=tk.RIGHT)
+      tk.Button(frm1, text='Refresh', command=lambda: self.open_(refresh=True)).pack(side=tk.RIGHT)
+    
+    # bottom status
+    # NOTE: do not know why the display order is messy...
+    frm3 = ttk.Label(wnd)
+    frm3.pack(side=tk.BOTTOM, anchor=tk.S, expand=tk.YES, fill=tk.X)
+    if True:
+      self.var_status = tk.StringVar(wnd)
+      tk.Label(frm3, textvariable=self.var_status).pack(anchor=tk.W)
+
+    # middel: plot
+    frm2 = ttk.Frame(wnd)
+    frm2.pack(expand=tk.YES, fill=tk.BOTH)
+    if True:
+      # left: control
+      frm21 = ttk.Frame(frm2)
+      frm21.pack(side=tk.LEFT, expand=tk.YES, fill=tk.BOTH)
+      if True:
+        # top: action
+        frm211 = ttk.Frame(frm21)
+        frm211.pack(side=tk.TOP, expand=tk.YES, fill=tk.X)
+        if True:
+          self.var_resr     = tk.BooleanVar(wnd, True)
+          self.var_resr_m   = tk.StringVar(wnd, 'realesr-animevideov3')
+          self.var_resr_r   = tk.IntVar(wnd, 2)
+          self.var_rife     = tk.BooleanVar(wnd, True)
+          self.var_rife_m   = tk.StringVar(wnd, 'rife-v4')
+          self.var_rife_r   = tk.IntVar(wnd, 2)
+          self.var_ffmpeg   = tk.BooleanVar(wnd, True)
+          self.var_ffmpeg_r = tk.IntVar(wnd, 20)
+          self.var_ffmpeg_f = tk.StringVar(wnd, 'mp4')
+
+          frm2111 = ttk.LabelFrame(frm211, text='Real-ESRGAN')
+          frm2111.pack(expand=tk.YES, fill=tk.X)
+          if True:
+            cb_m = ttk.Combobox(frm2111, text='model', values=list(RESR_MODELS.keys()), textvariable=self.var_resr_m, state='readonly', width=COMBOX_WIDTH)
+            cb_r = ttk.Combobox(frm2111, text='ratio', values=[], textvariable=self.var_resr_r, state='readonly', width=COMBOX_WIDTH1)
+            cb_m.grid(row=0, column=0, padx=2)
+            cb_r.grid(row=0, column=1, padx=2)
+            self.cb_resr = cb_r
+
+            def _cb_r_update():
+              values = RESR_MODELS[self.var_resr_m.get()]
+              cb_r.config(values=values)
+              if self.var_resr_r.get() not in values:
+                self.var_resr_r.set(values[0])
+              if len(values) == 1:
+                self.cb_resr.config(state=tk.DISABLED)
+              else:
+                self.cb_resr.config(state=tk.NORMAL)
+            cb_m.bind('<<ComboboxSelected>>', lambda evt: _cb_r_update())
+            _cb_r_update()
+
+          frm2112 = ttk.LabelFrame(frm211, text='RIFE')
+          frm2112.pack(expand=tk.YES, fill=tk.X)
+          if True:
+            cb = ttk.Combobox(frm2112, text='model', values=RIFE_MODELS, textvariable=self.var_rife_m, state='readonly', width=COMBOX_WIDTH)
+            et = ttk.Entry(frm2112, text='ratio', textvariable=self.var_rife_r, width=ENTRY_WIDTH)
+            cb.grid(row=0, column=0, padx=2)
+            et.grid(row=0, column=1, padx=2)
+            self.et_rife = et
+
+            def _et_update():
+              if self.var_rife_m.get() != 'rife-v4':
+                self.var_rife_r.set(2)
+                self.et_rife.config(state=tk.DISABLED)
+              else:
+                self.et_rife.config(state=tk.NORMAL)
+            cb.bind('<<ComboboxSelected>>', lambda evt: _et_update())
+            _et_update()
+
+          frm2113 = ttk.LabelFrame(frm211, text='FFmpeg')
+          frm2113.pack(expand=tk.YES, fill=tk.X)
+          if True:
+            cb = ttk.Combobox(frm2113, text='format', values=EXPORT_FMT, textvariable=self.var_ffmpeg_f, state='readonly', width=COMBOX_WIDTH)
+            et = ttk.Entry(frm2113, text='fps', textvariable=self.var_ffmpeg_r, width=ENTRY_WIDTH)
+            cb.grid(row=0, column=0, padx=2)
+            et.grid(row=0, column=1, padx=2)
+
+          frm2114 = ttk.Frame(frm211)
+          frm2114.pack(expand=tk.YES, fill=tk.X)
+          if True:
+            frm21141 = ttk.Frame(frm2114)
+            frm21141.pack(expand=tk.YES, fill=tk.X)
+            for i in range(3): frm21141.columnconfigure(i, weight=1)
+            if True:
+              ttk.Checkbutton(frm21141, text='resr',   variable=self.var_resr)  .grid(row=0, column=0, padx=0)
+              ttk.Checkbutton(frm21141, text='rife',   variable=self.var_rife)  .grid(row=0, column=1, padx=0)
+              ttk.Checkbutton(frm21141, text='ffmpeg', variable=self.var_ffmpeg).grid(row=0, column=2, padx=0)
+
+            btn = ttk.Button(frm2114, text='Run!', command=self.run)
+            btn.pack()
+            self.btn = btn
+
+        frm212 = ttk.LabelFrame(frm21, text='Travels')
+        frm212.pack(expand=tk.YES, fill=tk.BOTH)
+        if True:
+          self.var_ls = tk.StringVar()
+          sc = tk.Scrollbar(frm212, orient=tk.VERTICAL)
+          ls = tk.Listbox(frm212, listvariable=self.var_ls, selectmode=tk.BROWSE, yscrollcommand=sc.set, height=LIST_HEIGHT)
+          ls.bind('<<ListboxSelect>>', lambda evt: self._ls_change())
+          ls.pack(expand=tk.YES, fill=tk.BOTH)
+          sc.config(command=ls.yview)
+          sc.pack(side=tk.RIGHT, anchor=tk.E, expand=tk.YES, fill=tk.Y)
+          ls.bind('<Button-3>', menu_show)
+          self.ls = ls
+
+      # right: pv
+      frm22 = ttk.LabelFrame(frm2, text='Frames')
+      frm22.bind('<MouseWheel>', self._pv_change)
+      frm22.pack(side=tk.RIGHT, expand=tk.YES, fill=tk.BOTH)
+      if True:
+        # top
+        if True:
+          pv = ttk.Label(frm22, image=None)
+          pv.bind('<MouseWheel>', self._pv_change)
+          pv.bind('<Button-3>', menu_show)
+          pv.pack(anchor=tk.CENTER, expand=tk.YES, fill=tk.BOTH)
+          self.pv = pv
+
+        # bottom
+        if True:
+          self.var_fps_ip = tk.IntVar(wnd, 0)
+          sc = tk.Scale(frm22, orient=tk.HORIZONTAL, command=lambda _: self._pv_change(),
+                        from_=0, to=9, tickinterval=10, resolution=1, variable=self.var_fps_ip)
+          sc.bind('<MouseWheel>', self._pv_change)
+          sc.pack(anchor=tk.S, expand=tk.YES, fill=tk.X)
+          self.sc = sc
+
+  def _menu_open_dir(self):
+    try: startfile(Path(self.var_root_dp.get()) / self.cur_name)
+    except: print_exc()
+
+  def _menu_remove_dir(self):
+    idx: tuple = self.ls.curselection()
+    if not idx: return
+    name = self.ls.get(idx)
+    if name is None: return
+
+    dp = Path(self.var_root_dp.get()) / name
+    if name in self.cache:
+      cnt = len(self.cache[name])
+    else:
+      cnt = len([fp for fp in dp.iterdir() if fp.suffix.lower() in ['.png', '.jpg', '.jpeg']])
+    
+    if not tkmsg.askyesno('Remove', f'Confirm to remove folder "{name}" with {cnt} images?'):
+      return
+
+    try:
+      shutil.rmtree(str(dp))
+      self.ls.delete(idx)
+    except: print_exc()
+
+  def _mem_info_str(self, title='Mem'):
+    mem = self.p.memory_info()
+    return f'[{title}] rss: {mem.rss//2**20:.3f} MB, vms: {mem.vms//2**20:.3f} MB'
+
+  def mem_clear(self):
+    info1 = self._mem_info_str('Before')
+
+    to_del = set(self.cache.keys()) - {self.cur_name}
+    for name in to_del: del self.cache[name]
+    gc.collect()
+
+    info2 = self._mem_info_str('After')
+    tkmsg.showinfo('Meminfo', info1 + '\n' + info2)
+
+    self.cnt_pv_load = 0
+    self.var_status.set(self._mem_info_str())
+
+  def open_(self, root_dp:Path=None, refresh=False):
+    ''' Open a new travel root folder '''
+
+    if refresh: root_dp = self.var_root_dp.get()
+    if root_dp is None: root_dp = tkfdlg.askdirectory(initialdir=str(OUTPUT_PATH))
+    if not root_dp: return
+    if not Path(root_dp).exists():
+      tkmsg.showerror('Error', f'invalid path: {root_dp} not exist')
+      return
+
+    self.var_root_dp.set(root_dp)
+
+    dps = sorted([dp for dp in Path(root_dp).iterdir() if dp.is_dir()])
+    if len(dps) == 0: tkmsg.showerror('Error', 'No travels found!\Your root folder should be like <root_folder>/<travel_number>/*.png')
+
+    self.ls.selection_clear(0, tk.END)
+    self.var_ls.set([dp.name for dp in dps])
+
+    self.cache.clear() ; gc.collect()
+    self.ls.select_set(len(dps) - 1)
+    self.ls.yview_scroll(len(dps), 'units')
+    self._ls_change()
+
+  def _ls_change(self):
+    ''' Open a new travel id folder '''
+
+    idx: tuple = self.ls.curselection()
+    if not idx: return
+    name = self.ls.get(idx)
+    if name is None: return
+
+    self.cur_name = name
+    if name not in self.cache:
+      dp: Path = Path(self.var_root_dp.get()) / name
+      if dp.exists():
+        self.cache[name] = sorted([fp for fp in dp.iterdir() if fp.suffix.lower() in ['.png', '.jpg', '.jpeg'] and fp.stem != 'embryo'])
+      else:
+        self.ls.delete(idx)
+
+    n_imgs = len(self.cache[name])
+    self.sc.config(to=n_imgs-1)
+    try:    self.sc.config(tickinterval=n_imgs // (n_imgs / 10))
+    except: self.sc.config(tickinterval=1)
+
+    self.var_fps_ip.set(0)
+    self._pv_change()
+
+  def _pv_change(self, evt=None):
+    ''' Load a travel frame '''
+
+    if not self.cur_name: return
+
+    cache = self.cache[self.cur_name]
+    if not len(cache):
+      tkmsg.showinfo('Info', 'This folder is empty...')
+      return
+
+    idx = self.var_fps_ip.get()
+    if evt is not None:
+      offset = 1 if evt.delta < 0 else -1
+      idx = (idx + offset + len(cache)) % len(cache)
+      self.var_fps_ip.set(idx)
+
+    if isinstance(cache[idx], Path):
+      img = Image.open(cache[idx])
+      img.thumbnail((IMAGE_SIZE, IMAGE_SIZE), Image.LANCZOS)
+      cache[idx] = PhotoImage(img)
+
+      self.cnt_pv_load += 1
+      if self.cnt_pv_load >= MEMINFO_REFRESH:
+        self.cnt_pv_load = 0
+        self.var_status.set(self._mem_info_str())
+    
+    img = cache[idx]
+    self.pv.config(image=img)
+    self.pv.image = img
+
+  def run(self):
+    if self.is_running:
+      tkmsg.showerror('Error', 'Another task running at background, please wait before finish...')
+      return
+
+    def run_tasks(*args):
+      (
+        base_dp,
+        var_resr, var_resr_m, var_resr_r,
+        var_rife, var_rife_m, var_rife_r,
+        var_ffmpeg, var_ffmpeg_r, var_ffmpeg_f
+      ) = args
+
+      if not (0 <= var_rife_r < 8):
+        tkmsg.showerror('Error', f'rife_ratio is the interp ratio should be safe in range 0 ~ 4, but got {var_rife_r} :(')
+        return
+      if not (1 <= var_ffmpeg_r <= 60):
+        tkmsg.showerror('Error', f'fps should be safe in range 1 ~ 60, but got {var_ffmpeg_r} :(')
+        return
+
+      print('[Task] start') ; t = time()
+      try:
+        self.is_running = True
+        self.btn.config(state=tk.DISABLED, text='Running...')
+
+        if var_resr:
+          assert run_resr(var_resr_m, var_resr_r, base_dp, base_dp / 'resr')
+
+        if var_rife:
+          assert run_rife(var_rife_m, var_rife_r, base_dp / 'resr', base_dp / 'rife')
+
+        if var_ffmpeg:
+          dp: Path = base_dp / 'rife'
+          if dp.exists():
+            assert run_ffmpeg(var_ffmpeg_r, var_ffmpeg_f, base_dp / 'rife', base_dp)
+          else:
+            if tkmsg.askyesno('Warn', 'rife results not found, try synth from resr results?'):
+              assert run_ffmpeg(var_ffmpeg_r, var_ffmpeg_f, base_dp / 'resr', base_dp)
+
+        print(f'[Task] done ({time() - t:3f}s)')
+        r = tkmsg.askyesno('Ok', 'Task done! Open output folder?')
+        if r: startfile(base_dp)
+      except:
+        e = format_exc()
+        print(e)
+        print(f'[Task] faild ({time() - t:3f}s)')
+        tkmsg.showerror('Error', e)
+      finally:
+        self.is_running = False
+        self.btn.config(state=tk.NORMAL, text='Run!')
+
+    args = (
+      Path(self.var_root_dp.get()) / self.cur_name,
+      self.var_resr.get(),
+      self.var_resr_m.get(),
+      self.var_resr_r.get(),
+      self.var_rife.get(),
+      self.var_rife_m.get(),
+      self.var_rife_r.get(),
+      self.var_ffmpeg.get(),
+      self.var_ffmpeg_r.get(),
+      self.var_ffmpeg_f.get(),
+    )
+    Thread(target=run_tasks, args=args, daemon=True).start()
+    print(args)
+
+
+if __name__ == '__main__':
+  App()

postprocess-config.cmd.example

@@ -0,0 +1,43 @@
+@REM Configs for postprocess.cmd
+@ECHO OFF
+
+REM Real-ESRGAN model ckpt
+REM   string, [realesr-animevideov3, realesrgan-x4plus-anime, realesrgan-x4plus]
+REM   default: realesr-animevideov3
+SET RESR_MODEL=realesr-animevideov3
+
+REM image upscale rate
+REM   int, [2, 3, 4]
+REM   default: 2
+SET RESR_UPSCALE=2
+
+REM RIFE model ckpt
+REM   string, [rife-v4.6, rife-v4, rife-v2.3, rife-anime, ...]
+REM   default: rife-v4
+SET RIFE_MODEL=rife-v4
+
+REM interpolated frame count
+REM   int, 0 means n_images x 2
+REM   default: 0
+SET RIFE_INTERP=0
+
+REM rendered video fps, higher value requires more interpolations
+REM   int, 12 ~ 60 should be fine
+REM   default: 20 (to match the default fps of prompt-travel)
+SET FPS=20
+
+
+REM time count down before task start (in seconds)
+REM   int, non-negative
+REM   default: 5
+SET WAIT_BEFORE_START=5
+
+REM auto launch explorer and locate the generated file when done
+REM   boolean, [0, 1]
+REM   default: 1
+SET EXPLORER_FLAG=1
+
+REM clean all cache files when done, saving disk usage
+REM   boolean, [0, 1]
+REM   default: 0
+SET CLEAN_FLAG=0

postprocess.cmd

@@ -0,0 +1,164 @@
+@REM Handy script for post-process pipeline
+@ECHO OFF
+SETLOCAL
+
+TITLE Post-processing for prompt-travel...
+
+REM remeber base path and script name
+SET _=%~dp0
+SET $=%~nx0
+SHIFT
+
+REM init configs or make default
+SET CONFIG_FILE=%_%postprocess-config.cmd
+IF EXIST %CONFIG_FILE% GOTO skip_init_cfg
+COPY %CONFIG_FILE%.example %CONFIG_FILE%
+IF ERRORLEVEL 1 GOTO die
+:skip_init_cfg
+
+REM load configs
+CALL %CONFIG_FILE%
+IF ERRORLEVEL 1 GOTO die
+
+REM assert required arguments
+IF /I "%~0"=="-c" (
+  SET CLEAN_FLAG=1
+  SHIFT
+)
+SET IMAGE_FOLDER=%~0
+SHIFT
+
+REM show help
+IF NOT EXIST "%IMAGE_FOLDER%" (
+  ECHO Usage: %$% [-c] ^<image_folder^> [upscale] [interp] [fps] [resr_model] [rife_model]
+  ECHO    -c            clean cache data when done
+  ECHO    upscale       image upsampling rate ^(default: %RESR_UPSCALE%^)
+  ECHO    interp        interpolated video frame count ^(default: %RIFE_INTERP%^)
+  ECHO    fps           rendered video frame rate ^(default: %FPS%^)
+  ECHO    resr_model    Real-ESRGAN model checkpoint name ^(default: %RESR_MODEL%^)
+  ECHO    rife_model    RIFE model checkpoint name ^(default: %RIFE_MODEL%^)
+  ECHO.
+  ECHO    e.g. %$% D:\images
+  ECHO         %$% -c D:\images
+  ECHO         %$% D:\images 2 0
+  ECHO         %$% D:\images 4 120 24
+  ECHO         %$% D:\images 4 0 24 realesr-animevideov3 rife-v2.3
+  ECHO    note:
+  ECHO         ^<args^> arguments are required
+  ECHO         ^[args^] arguments are optional
+  ECHO.
+  GOTO :end
+)
+
+REM override optional arguments by command line
+IF NOT "%~0"=="" (
+  SET RESR_UPSCALE=%~0
+  SHIFT
+)
+IF NOT "%~0"=="" (
+  SET RIFE_INTERP=%~0
+  SHIFT
+)
+IF NOT "%~0"=="" (
+  SET FPS=%~0
+  SHIFT
+)
+IF NOT "%~0"=="" (
+  SET RESR_MODEL=%~0
+  SHIFT
+)
+IF NOT "%~0"=="" (
+  SET RIFE_MODEL=%~0
+  SHIFT
+)
+
+REM prepare paths
+SET TOOL_HOME=%_%tools
+SET RESR_HOME=%TOOL_HOME%\realesrgan-ncnn-vulkan
+SET RIFE_HOME=%TOOL_HOME%\rife-ncnn-vulkan
+SET FFMPEG_HOME=%TOOL_HOME%\ffmpeg
+
+SET BBOX_BIN=busybox.exe
+SET RESR_BIN=realesrgan-ncnn-vulkan.exe
+SET RIFE_BIN=rife-ncnn-vulkan.exe
+SET FFMPEG_BIN=ffmpeg.exe
+
+PATH %TOOL_HOME%;%PATH%
+PATH %RESR_HOME%;%PATH%
+PATH %RIFE_HOME%;%PATH%
+PATH %FFMPEG_HOME%\bin;%FFMPEG_HOME%;%PATH%
+
+SET RESR_FOLDER=%IMAGE_FOLDER%\resr
+SET RIFE_FOLDER=%IMAGE_FOLDER%\rife
+SET OUT_FILE=%IMAGE_FOLDER%\synth.mp4
+
+REM show configs for debug
+ECHO ==================================================
+ECHO RESR_MODEL   = %RESR_MODEL%
+ECHO RESR_UPSCALE = %RESR_UPSCALE%
+ECHO RIFE_MODEL   = %RIFE_MODEL%
+ECHO RIFE_INTERP  = %RIFE_INTERP%
+ECHO FPS          = %FPS%
+ECHO RESR_FOLDER  = %RESR_FOLDER%
+ECHO RIFE_FOLDER  = %RIFE_FOLDER%
+ECHO OUT_FILE     = %OUT_FILE%
+ECHO.
+
+ECHO ^>^> wait for %WAIT_BEFORE_START% seconds before start...
+%BBOX_BIN% sleep %WAIT_BEFORE_START%
+IF ERRORLEVEL 1 GOTO die
+ECHO ^>^> start processing!
+
+REM start processing
+ECHO ==================================================
+
+ECHO [1/3] image super-resolution
+IF EXIST %RESR_FOLDER% GOTO skip_resr
+MKDIR %RESR_FOLDER%
+%RESR_BIN% -v -s %RESR_UPSCALE% -n %RESR_MODEL% -i %IMAGE_FOLDER% -o %RESR_FOLDER%
+IF ERRORLEVEL 1 GOTO die
+:skip_resr
+
+ECHO ==================================================
+
+ECHO [2/3] video frame-interpolation
+IF EXIST %RIFE_FOLDER% GOTO skip_rife
+MKDIR %RIFE_FOLDER%
+SET NFRAMES=%RESR_FOLDER%
+
+%RIFE_BIN% -v -n %RIFE_INTERP% -m %RIFE_MODEL% -i %RESR_FOLDER% -o %RIFE_FOLDER%
+IF ERRORLEVEL 1 GOTO die
+:skip_rife
+
+ECHO ==================================================
+
+ECHO [3/3] render video
+%FFMPEG_BIN% -y -framerate %FPS% -i %RIFE_FOLDER%\%%08d.png -crf 20 -c:v libx264 -pix_fmt yuv420p %OUT_FILE%
+IF ERRORLEVEL 1 GOTO die
+
+ECHO ==================================================
+
+REM clean cache
+IF "%CLEAN_FLAG%"=="1" (
+  RMDIR /S /Q %RESR_FOLDER%
+  RMDIR /S /Q %RIFE_FOLDER%
+)
+
+REM finished
+ECHO ^>^> file saved to %OUT_FILE%
+IF "%EXPLORER_FLAG%"=="1" (
+  explorer.exe /e,/select,%OUT_FILE%
+)
+
+ECHO ^>^> Done!
+ECHO.
+GOTO :end
+
+REM error handle
+:die
+ECHO ^<^< Error!
+ECHO ^<^< errorlevel: %ERRORLEVEL%
+ECHO.
+
+:end
+PAUSE

requirements.txt

@@ -0,0 +1,6 @@
+# webui script
+moviepy
+
+# postprocessor (GUI)
+psutil
+Pillow

scripts/controlnet_travel.py

@@ -0,0 +1,913 @@
+# This extension works with [Mikubill/sd-webui-controlnet](https://github.com/Mikubill/sd-webui-controlnet)
+# version: v1.1.229
+
+LOG_PREFIX = '[ControlNet-Travel]'
+
+# ↓↓↓ EXIT EARLY IF EXTERNAL REPOSITORY NOT FOUND ↓↓↓
+
+CTRLNET_REPO_NAME = 'sdcontrol'
+if 'externel repo sanity check':
+    from pathlib import Path
+    from modules.scripts import basedir
+    from traceback import print_exc
+
+    ME_PATH = Path(basedir())
+    CTRLNET_PATH = ME_PATH.parent / 'sdcontrol'
+
+    controlnet_found = False
+    try:
+        import sys ; sys.path.append(str(CTRLNET_PATH))
+        #from scripts.controlnet import Script as ControlNetScript  # NOTE: this will mess up the import order
+        from scripts.external_code import ControlNetUnit
+        from scripts.hook import UNetModel, UnetHook, ControlParams
+        from scripts.hook import *
+
+        controlnet_found = True
+        print(f'{LOG_PREFIX} extension {CTRLNET_REPO_NAME} found, ControlNet-Travel loaded :)')
+    except ImportError:
+        print(f'{LOG_PREFIX} extension {CTRLNET_REPO_NAME} not found, ControlNet-Travel ignored :(')
+        exit(0)
+    except:
+        print_exc()
+        exit(0)
+
+# ↑↑↑ EXIT EARLY IF EXTERNAL REPOSITORY NOT FOUND ↑↑↑
+
+
+import sys
+from PIL import Image
+
+from ldm.models.diffusion.ddpm import LatentDiffusion
+from modules import shared, devices, lowvram
+from modules.processing import StableDiffusionProcessing as Processing
+
+from scripts.prompt_travel import *
+from manager import run_cmd
+
+class InterpMethod(Enum):
+    LINEAR = 'linear (weight sum)'
+    RIFE   = 'rife (optical flow)'
+
+if 'consts':
+    __ = lambda key, value=None: opts.data.get(f'customscript/controlnet_travel.py/txt2img/{key}/value', value)
+
+
+    LABEL_CTRLNET_REF_DIR   = 'Reference image folder (one ref image per stage :)'
+    LABEL_INTERP_METH       = 'Interpolate method'
+    LABEL_SKIP_FUSE         = 'Ext. skip latent fusion'
+    LABEL_DEBUG_RIFE        = 'Save RIFE intermediates'
+
+    DEFAULT_STEPS           = 10
+    DEFAULT_CTRLNET_REF_DIR = str(ME_PATH / 'img' / 'ref_ctrlnet')
+    DEFAULT_INTERP_METH     = __(LABEL_INTERP_METH, InterpMethod.LINEAR.value)
+    DEFAULT_SKIP_FUSE       = __(LABEL_SKIP_FUSE, False)
+    DEFAULT_DEBUG_RIFE      = __(LABEL_DEBUG_RIFE, False)
+
+    CHOICES_INTERP_METH     = [x.value for x in InterpMethod]
+
+if 'vars':
+    skip_fuse_plan:       List[bool]         = []   # n_blocks (13)
+
+    interp_alpha:         float              = 0.0
+    interp_ip:            int                = 0    # 0 ~ n_sampling_step-1
+    from_hint_cond:       List[Tensor]       = []   # n_contrlnet_set
+    to_hint_cond:         List[Tensor]       = []
+    mid_hint_cond:        List[Tensor]       = []
+    from_control_tensors: List[List[Tensor]] = []   # n_sampling_step x n_blocks
+    to_control_tensors:   List[List[Tensor]] = []
+
+    caches: List[list] = [from_hint_cond, to_hint_cond, mid_hint_cond, from_control_tensors, to_control_tensors]
+
+
+# ↓↓↓ the following is modified from 'sd-webui-controlnet/scripts/hook.py' ↓↓↓
+
+def hook_hijack(self:UnetHook, model:UNetModel, sd_ldm:LatentDiffusion, control_params:List[ControlParams], process:Processing):
+    self.model = model
+    self.sd_ldm = sd_ldm
+    self.control_params = control_params
+
+    outer = self
+
+    def process_sample(*args, **kwargs):
+        # ControlNet must know whether a prompt is conditional prompt (positive prompt) or unconditional conditioning prompt (negative prompt).
+        # You can use the hook.py's `mark_prompt_context` to mark the prompts that will be seen by ControlNet.
+        # Let us say XXX is a MulticondLearnedConditioning or a ComposableScheduledPromptConditioning or a ScheduledPromptConditioning or a list of these components,
+        # if XXX is a positive prompt, you should call mark_prompt_context(XXX, positive=True)
+        # if XXX is a negative prompt, you should call mark_prompt_context(XXX, positive=False)
+        # After you mark the prompts, the ControlNet will know which prompt is cond/uncond and works as expected.
+        # After you mark the prompts, the mismatch errors will disappear.
+        mark_prompt_context(kwargs.get('conditioning', []), positive=True)
+        mark_prompt_context(kwargs.get('unconditional_conditioning', []), positive=False)
+        mark_prompt_context(getattr(process, 'hr_c', []), positive=True)
+        mark_prompt_context(getattr(process, 'hr_uc', []), positive=False)
+        return process.sample_before_CN_hack(*args, **kwargs)
+
+    # NOTE: ↓↓↓ only hack this method ↓↓↓
+    def forward(self:UNetModel, x:Tensor, timesteps:Tensor=None, context:Tensor=None, **kwargs):
+        total_controlnet_embedding = [0.0] * 13
+        total_t2i_adapter_embedding = [0.0] * 4
+        require_inpaint_hijack = False
+        is_in_high_res_fix = False
+        batch_size = int(x.shape[0])
+
+        # NOTE: declare globals
+        global from_hint_cond, to_hint_cond, from_control_tensors, to_control_tensors, mid_hint_cond, interp_alpha, interp_ip
+        x: Tensor           # [1, 4, 64, 64]
+        timesteps: Tensor   # [1]
+        context: Tensor     # [1, 78, 768]
+        kwargs: dict        # {}
+
+        # Handle cond-uncond marker
+        cond_mark, outer.current_uc_indices, context = unmark_prompt_context(context)
+        # logger.info(str(cond_mark[:, 0, 0, 0].detach().cpu().numpy().tolist()) + ' - ' + str(outer.current_uc_indices))
+
+        # High-res fix
+        for param in outer.control_params:
+            # select which hint_cond to use
+            if param.used_hint_cond is None:
+                param.used_hint_cond = param.hint_cond      # NOTE: input hint cond tensor, [1, 3, 512, 512]
+                param.used_hint_cond_latent = None
+                param.used_hint_inpaint_hijack = None
+
+            # has high-res fix
+            if param.hr_hint_cond is not None and x.ndim == 4 and param.hint_cond.ndim == 4 and param.hr_hint_cond.ndim == 4:
+                _, _, h_lr, w_lr = param.hint_cond.shape
+                _, _, h_hr, w_hr = param.hr_hint_cond.shape
+                _, _, h, w = x.shape
+                h, w = h * 8, w * 8
+                if abs(h - h_lr) < abs(h - h_hr):
+                    is_in_high_res_fix = False
+                    if param.used_hint_cond is not param.hint_cond:
+                        param.used_hint_cond = param.hint_cond
+                        param.used_hint_cond_latent = None
+                        param.used_hint_inpaint_hijack = None
+                else:
+                    is_in_high_res_fix = True
+                    if param.used_hint_cond is not param.hr_hint_cond:
+                        param.used_hint_cond = param.hr_hint_cond
+                        param.used_hint_cond_latent = None
+                        param.used_hint_inpaint_hijack = None
+
+        # NOTE: hint shallow fusion, overwrite param.used_hint_cond
+        for i, param in enumerate(outer.control_params):
+            if interp_alpha == 0.0:     # collect hind_cond on key frames
+                if len(to_hint_cond) < len(outer.control_params):
+                    to_hint_cond.append(param.used_hint_cond.clone().detach().cpu())
+            else:                       # interp with cached hind_cond
+                param.used_hint_cond = mid_hint_cond[i].to(x.device)
+
+        # Convert control image to latent
+        for param in outer.control_params:
+            if param.used_hint_cond_latent is not None:
+                continue
+            if param.control_model_type not in [ControlModelType.AttentionInjection] \
+                    and 'colorfix' not in param.preprocessor['name'] \
+                    and 'inpaint_only' not in param.preprocessor['name']:
+                continue
+            param.used_hint_cond_latent = outer.call_vae_using_process(process, param.used_hint_cond, batch_size=batch_size)
+                
+        # handle prompt token control
+        for param in outer.control_params:
+            if param.guidance_stopped:
+                continue
+
+            if param.control_model_type not in [ControlModelType.T2I_StyleAdapter]:
+                continue
+
+            param.control_model.to(devices.get_device_for("controlnet"))
+            control = param.control_model(x=x, hint=param.used_hint_cond, timesteps=timesteps, context=context)
+            control = torch.cat([control.clone() for _ in range(batch_size)], dim=0)
+            control *= param.weight
+            control *= cond_mark[:, :, :, 0]
+            context = torch.cat([context, control.clone()], dim=1)
+
+        # handle ControlNet / T2I_Adapter
+        for param in outer.control_params:
+            if param.guidance_stopped:
+                continue
+
+            if param.control_model_type not in [ControlModelType.ControlNet, ControlModelType.T2I_Adapter]:
+                continue
+
+            param.control_model.to(devices.get_device_for("controlnet"))
+            # inpaint model workaround
+            x_in = x
+            control_model = param.control_model.control_model
+
+            if param.control_model_type == ControlModelType.ControlNet:
+                if x.shape[1] != control_model.input_blocks[0][0].in_channels and x.shape[1] == 9:
+                    # inpaint_model: 4 data + 4 downscaled image + 1 mask
+                    x_in = x[:, :4, ...]
+                    require_inpaint_hijack = True
+
+            assert param.used_hint_cond is not None, f"Controlnet is enabled but no input image is given"
+
+            hint = param.used_hint_cond
+
+            # ControlNet inpaint protocol
+            if hint.shape[1] == 4:
+                c = hint[:, 0:3, :, :]
+                m = hint[:, 3:4, :, :]
+                m = (m > 0.5).float()
+                hint = c * (1 - m) - m
+
+            # NOTE: len(control) == 13, control[i]:Tensor
+            control = param.control_model(x=x_in, hint=hint, timesteps=timesteps, context=context)
+            control_scales = ([param.weight] * 13)
+
+            if outer.lowvram:
+                param.control_model.to("cpu")
+
+            if param.cfg_injection or param.global_average_pooling:
+                if param.control_model_type == ControlModelType.T2I_Adapter:
+                    control = [torch.cat([c.clone() for _ in range(batch_size)], dim=0) for c in control]
+                control = [c * cond_mark for c in control]
+
+            high_res_fix_forced_soft_injection = False
+
+            if is_in_high_res_fix:
+                if 'canny' in param.preprocessor['name']:
+                    high_res_fix_forced_soft_injection = True
+                if 'mlsd' in param.preprocessor['name']:
+                    high_res_fix_forced_soft_injection = True
+
+            # if high_res_fix_forced_soft_injection:
+            #     logger.info('[ControlNet] Forced soft_injection in high_res_fix in enabled.')
+
+            if param.soft_injection or high_res_fix_forced_soft_injection:
+                # important! use the soft weights with high-res fix can significantly reduce artifacts.
+                if param.control_model_type == ControlModelType.T2I_Adapter:
+                    control_scales = [param.weight * x for x in (0.25, 0.62, 0.825, 1.0)]
+                elif param.control_model_type == ControlModelType.ControlNet:
+                    control_scales = [param.weight * (0.825 ** float(12 - i)) for i in range(13)]
+
+            if param.advanced_weighting is not None:
+                control_scales = param.advanced_weighting
+
+            control = [c * scale for c, scale in zip(control, control_scales)]
+            if param.global_average_pooling:
+                control = [torch.mean(c, dim=(2, 3), keepdim=True) for c in control]
+
+            for idx, item in enumerate(control):
+                target = None
+                if param.control_model_type == ControlModelType.ControlNet:
+                    target = total_controlnet_embedding
+                if param.control_model_type == ControlModelType.T2I_Adapter:
+                    target = total_t2i_adapter_embedding
+                if target is not None:
+                    target[idx] = item + target[idx]
+
+        # Replace x_t to support inpaint models
+        for param in outer.control_params:
+            if param.used_hint_cond.shape[1] != 4:
+                continue
+            if x.shape[1] != 9:
+                continue
+            if param.used_hint_inpaint_hijack is None:
+                mask_pixel = param.used_hint_cond[:, 3:4, :, :]
+                image_pixel = param.used_hint_cond[:, 0:3, :, :]
+                mask_pixel = (mask_pixel > 0.5).to(mask_pixel.dtype)
+                masked_latent = outer.call_vae_using_process(process, image_pixel, batch_size, mask=mask_pixel)
+                mask_latent = torch.nn.functional.max_pool2d(mask_pixel, (8, 8))
+                if mask_latent.shape[0] != batch_size:
+                    mask_latent = torch.cat([mask_latent.clone() for _ in range(batch_size)], dim=0)
+                param.used_hint_inpaint_hijack = torch.cat([mask_latent, masked_latent], dim=1)
+                param.used_hint_inpaint_hijack.to(x.dtype).to(x.device)
+            x = torch.cat([x[:, :4, :, :], param.used_hint_inpaint_hijack], dim=1)
+
+        # A1111 fix for medvram.
+        if shared.cmd_opts.medvram:
+            try:
+                # Trigger the register_forward_pre_hook
+                outer.sd_ldm.model()
+            except:
+                pass
+
+        # Clear attention and AdaIn cache
+        for module in outer.attn_module_list:
+            module.bank = []
+            module.style_cfgs = []
+        for module in outer.gn_module_list:
+            module.mean_bank = []
+            module.var_bank = []
+            module.style_cfgs = []
+
+        # Handle attention and AdaIn control
+        for param in outer.control_params:
+            if param.guidance_stopped:
+                continue
+
+            if param.used_hint_cond_latent is None:
+                continue
+
+            if param.control_model_type not in [ControlModelType.AttentionInjection]:
+                continue
+
+            ref_xt = outer.sd_ldm.q_sample(param.used_hint_cond_latent, torch.round(timesteps.float()).long())
+
+            # Inpaint Hijack
+            if x.shape[1] == 9:
+                ref_xt = torch.cat([
+                    ref_xt,
+                    torch.zeros_like(ref_xt)[:, 0:1, :, :],
+                    param.used_hint_cond_latent
+                ], dim=1)
+
+            outer.current_style_fidelity = float(param.preprocessor['threshold_a'])
+            outer.current_style_fidelity = max(0.0, min(1.0, outer.current_style_fidelity))
+
+            if param.cfg_injection:
+                outer.current_style_fidelity = 1.0
+            elif param.soft_injection or is_in_high_res_fix:
+                outer.current_style_fidelity = 0.0
+
+            control_name = param.preprocessor['name']
+
+            if control_name in ['reference_only', 'reference_adain+attn']:
+                outer.attention_auto_machine = AutoMachine.Write
+                outer.attention_auto_machine_weight = param.weight
+
+            if control_name in ['reference_adain', 'reference_adain+attn']:
+                outer.gn_auto_machine = AutoMachine.Write
+                outer.gn_auto_machine_weight = param.weight
+
+            outer.original_forward(
+                x=ref_xt.to(devices.dtype_unet),
+                timesteps=timesteps.to(devices.dtype_unet),
+                context=context.to(devices.dtype_unet)
+            )
+
+            outer.attention_auto_machine = AutoMachine.Read
+            outer.gn_auto_machine = AutoMachine.Read
+
+        # NOTE: hint latent fusion, overwrite control tensors
+        total_control = total_controlnet_embedding
+        if interp_alpha == 0.0:     # collect control tensors on key frames
+            tensors: List[Tensor] = []
+            for i, t in enumerate(total_control):
+                if len(skip_fuse_plan) and skip_fuse_plan[i]:
+                    tensors.append(None)
+                else:
+                    tensors.append(t.clone().detach().cpu())
+            to_control_tensors.append(tensors)
+        else:                       # interp with cached control tensors
+            device = total_control[0].device
+            for i, (ctrlA, ctrlB) in enumerate(zip(from_control_tensors[interp_ip], to_control_tensors[interp_ip])):
+                if ctrlA is not None and ctrlB is not None:
+                    ctrlC = weighted_sum(ctrlA.to(device), ctrlB.to(device), interp_alpha)
+                    #print('  ctrl diff:', (ctrlC - total_control[i]).abs().mean().item())
+                    total_control[i].data = ctrlC
+            interp_ip += 1
+        
+        # NOTE: warn on T2I adapter
+        if total_t2i_adapter_embedding[0] != 0:
+            print(f'{LOG_PREFIX} warn: currently t2i_adapter is not supported. if you wanna this, put a feature request on Kahsolt/stable-diffusion-webui-prompt-travel')
+
+        # U-Net Encoder
+        hs = []
+        with th.no_grad():
+            t_emb = cond_cast_unet(timestep_embedding(timesteps, self.model_channels, repeat_only=False))
+            emb = self.time_embed(t_emb)
+            h = x.type(self.dtype)
+            for i, module in enumerate(self.input_blocks):
+                h = module(h, emb, context)
+
+                if (i + 1) % 3 == 0:
+                    h = aligned_adding(h, total_t2i_adapter_embedding.pop(0), require_inpaint_hijack)
+
+                hs.append(h)
+            h = self.middle_block(h, emb, context)
+
+        # U-Net Middle Block
+        h = aligned_adding(h, total_controlnet_embedding.pop(), require_inpaint_hijack)
+
+        # U-Net Decoder
+        for i, module in enumerate(self.output_blocks):
+            h = th.cat([h, aligned_adding(hs.pop(), total_controlnet_embedding.pop(), require_inpaint_hijack)], dim=1)
+            h = module(h, emb, context)
+
+        # U-Net Output
+        h = h.type(x.dtype)
+        h = self.out(h)
+
+        # Post-processing for color fix
+        for param in outer.control_params:
+            if param.used_hint_cond_latent is None:
+                continue
+            if 'colorfix' not in param.preprocessor['name']:
+                continue
+
+            k = int(param.preprocessor['threshold_a'])
+            if is_in_high_res_fix:
+                k *= 2
+
+            # Inpaint hijack
+            xt = x[:, :4, :, :]
+
+            x0_origin = param.used_hint_cond_latent
+            t = torch.round(timesteps.float()).long()
+            x0_prd = predict_start_from_noise(outer.sd_ldm, xt, t, h)
+            x0 = x0_prd - blur(x0_prd, k) + blur(x0_origin, k)
+
+            if '+sharp' in param.preprocessor['name']:
+                detail_weight = float(param.preprocessor['threshold_b']) * 0.01
+                neg = detail_weight * blur(x0, k) + (1 - detail_weight) * x0
+                x0 = cond_mark * x0 + (1 - cond_mark) * neg
+
+            eps_prd = predict_noise_from_start(outer.sd_ldm, xt, t, x0)
+
+            w = max(0.0, min(1.0, float(param.weight)))
+            h = eps_prd * w + h * (1 - w)
+
+        # Post-processing for restore
+        for param in outer.control_params:
+            if param.used_hint_cond_latent is None:
+                continue
+            if 'inpaint_only' not in param.preprocessor['name']:
+                continue
+            if param.used_hint_cond.shape[1] != 4:
+                continue
+
+            # Inpaint hijack
+            xt = x[:, :4, :, :]
+
+            mask = param.used_hint_cond[:, 3:4, :, :]
+            mask = torch.nn.functional.max_pool2d(mask, (10, 10), stride=(8, 8), padding=1)
+
+            x0_origin = param.used_hint_cond_latent
+            t = torch.round(timesteps.float()).long()
+            x0_prd = predict_start_from_noise(outer.sd_ldm, xt, t, h)
+            x0 = x0_prd * mask + x0_origin * (1 - mask)
+            eps_prd = predict_noise_from_start(outer.sd_ldm, xt, t, x0)
+
+            w = max(0.0, min(1.0, float(param.weight)))
+            h = eps_prd * w + h * (1 - w)
+
+        return h
+
+    def forward_webui(*args, **kwargs):
+        # webui will handle other compoments 
+        try:
+            if shared.cmd_opts.lowvram:
+                lowvram.send_everything_to_cpu()
+
+            return forward(*args, **kwargs)
+        finally:
+            if self.lowvram:
+                for param in self.control_params:
+                    if isinstance(param.control_model, torch.nn.Module):
+                        param.control_model.to("cpu")
+
+    def hacked_basic_transformer_inner_forward(self, x, context=None):
+        x_norm1 = self.norm1(x)
+        self_attn1 = None
+        if self.disable_self_attn:
+            # Do not use self-attention
+            self_attn1 = self.attn1(x_norm1, context=context)
+        else:
+            # Use self-attention
+            self_attention_context = x_norm1
+            if outer.attention_auto_machine == AutoMachine.Write:
+                if outer.attention_auto_machine_weight > self.attn_weight:
+                    self.bank.append(self_attention_context.detach().clone())
+                    self.style_cfgs.append(outer.current_style_fidelity)
+            if outer.attention_auto_machine == AutoMachine.Read:
+                if len(self.bank) > 0:
+                    style_cfg = sum(self.style_cfgs) / float(len(self.style_cfgs))
+                    self_attn1_uc = self.attn1(x_norm1, context=torch.cat([self_attention_context] + self.bank, dim=1))
+                    self_attn1_c = self_attn1_uc.clone()
+                    if len(outer.current_uc_indices) > 0 and style_cfg > 1e-5:
+                        self_attn1_c[outer.current_uc_indices] = self.attn1(
+                            x_norm1[outer.current_uc_indices],
+                            context=self_attention_context[outer.current_uc_indices])
+                    self_attn1 = style_cfg * self_attn1_c + (1.0 - style_cfg) * self_attn1_uc
+                self.bank = []
+                self.style_cfgs = []
+            if self_attn1 is None:
+                self_attn1 = self.attn1(x_norm1, context=self_attention_context)
+
+        x = self_attn1.to(x.dtype) + x
+        x = self.attn2(self.norm2(x), context=context) + x
+        x = self.ff(self.norm3(x)) + x
+        return x
+
+    def hacked_group_norm_forward(self, *args, **kwargs):
+        eps = 1e-6
+        x = self.original_forward(*args, **kwargs)
+        y = None
+        if outer.gn_auto_machine == AutoMachine.Write:
+            if outer.gn_auto_machine_weight > self.gn_weight:
+                var, mean = torch.var_mean(x, dim=(2, 3), keepdim=True, correction=0)
+                self.mean_bank.append(mean)
+                self.var_bank.append(var)
+                self.style_cfgs.append(outer.current_style_fidelity)
+        if outer.gn_auto_machine == AutoMachine.Read:
+            if len(self.mean_bank) > 0 and len(self.var_bank) > 0:
+                style_cfg = sum(self.style_cfgs) / float(len(self.style_cfgs))
+                var, mean = torch.var_mean(x, dim=(2, 3), keepdim=True, correction=0)
+                std = torch.maximum(var, torch.zeros_like(var) + eps) ** 0.5
+                mean_acc = sum(self.mean_bank) / float(len(self.mean_bank))
+                var_acc = sum(self.var_bank) / float(len(self.var_bank))
+                std_acc = torch.maximum(var_acc, torch.zeros_like(var_acc) + eps) ** 0.5
+                y_uc = (((x - mean) / std) * std_acc) + mean_acc
+                y_c = y_uc.clone()
+                if len(outer.current_uc_indices) > 0 and style_cfg > 1e-5:
+                    y_c[outer.current_uc_indices] = x.to(y_c.dtype)[outer.current_uc_indices]
+                y = style_cfg * y_c + (1.0 - style_cfg) * y_uc
+            self.mean_bank = []
+            self.var_bank = []
+            self.style_cfgs = []
+        if y is None:
+            y = x
+        return y.to(x.dtype)
+
+    if getattr(process, 'sample_before_CN_hack', None) is None:
+        process.sample_before_CN_hack = process.sample
+    process.sample = process_sample
+
+    model._original_forward = model.forward
+    outer.original_forward = model.forward
+    model.forward = forward_webui.__get__(model, UNetModel)
+
+    all_modules = torch_dfs(model)
+
+    attn_modules = [module for module in all_modules if isinstance(module, BasicTransformerBlock)]
+    attn_modules = sorted(attn_modules, key=lambda x: - x.norm1.normalized_shape[0])
+
+    for i, module in enumerate(attn_modules):
+        if getattr(module, '_original_inner_forward', None) is None:
+            module._original_inner_forward = module._forward
+        module._forward = hacked_basic_transformer_inner_forward.__get__(module, BasicTransformerBlock)
+        module.bank = []
+        module.style_cfgs = []
+        module.attn_weight = float(i) / float(len(attn_modules))
+
+    gn_modules = [model.middle_block]
+    model.middle_block.gn_weight = 0
+
+    input_block_indices = [4, 5, 7, 8, 10, 11]
+    for w, i in enumerate(input_block_indices):
+        module = model.input_blocks[i]
+        module.gn_weight = 1.0 - float(w) / float(len(input_block_indices))
+        gn_modules.append(module)
+
+    output_block_indices = [0, 1, 2, 3, 4, 5, 6, 7]
+    for w, i in enumerate(output_block_indices):
+        module = model.output_blocks[i]
+        module.gn_weight = float(w) / float(len(output_block_indices))
+        gn_modules.append(module)
+
+    for i, module in enumerate(gn_modules):
+        if getattr(module, 'original_forward', None) is None:
+            module.original_forward = module.forward
+        module.forward = hacked_group_norm_forward.__get__(module, torch.nn.Module)
+        module.mean_bank = []
+        module.var_bank = []
+        module.style_cfgs = []
+        module.gn_weight *= 2
+
+    outer.attn_module_list = attn_modules
+    outer.gn_module_list = gn_modules
+
+    scripts.script_callbacks.on_cfg_denoiser(self.guidance_schedule_handler)
+
+# ↑↑↑ the above is modified from 'sd-webui-controlnet/scripts/hook.py' ↑↑↑
+
+def reset_cuda():
+    devices.torch_gc()
+    import gc; gc.collect()
+
+    try:
+        import os
+        import psutil
+        mem = psutil.Process(os.getpid()).memory_info()
+        print(f'[Mem] rss: {mem.rss/2**30:.3f} GB, vms: {mem.vms/2**30:.3f} GB')
+        from modules.shared import mem_mon as vram_mon
+        free, total = vram_mon.cuda_mem_get_info()
+        print(f'[VRAM] free: {free/2**30:.3f} GB, total: {total/2**30:.3f} GB')
+    except:
+        pass
+
+
+class Script(scripts.Script):
+
+    def title(self):
+        return 'ControlNet Travel'
+
+    def describe(self):
+        return 'Travel from one controlnet hint condition to another in the tensor space.'
+
+    def show(self, is_img2img):
+        return controlnet_found
+
+    def ui(self, is_img2img):
+        with gr.Row(variant='compact'):
+            interp_meth = gr.Dropdown(label=LABEL_INTERP_METH, value=lambda: DEFAULT_INTERP_METH, choices=CHOICES_INTERP_METH)
+            steps       = gr.Text    (label=LABEL_STEPS,       value=lambda: DEFAULT_STEPS,       max_lines=1)
+            
+            reset = gr.Button(value='Reset Cuda', variant='tool')
+            reset.click(fn=reset_cuda, show_progress=False)
+
+        with gr.Row(variant='compact'):
+            ctrlnet_ref_dir = gr.Text(label=LABEL_CTRLNET_REF_DIR, value=lambda: DEFAULT_CTRLNET_REF_DIR, max_lines=1)
+
+        with gr.Group(visible=DEFAULT_SKIP_FUSE) as tab_ext_skip_fuse:
+            with gr.Row(variant='compact'):
+                skip_in_0  = gr.Checkbox(label='in_0')
+                skip_in_3  = gr.Checkbox(label='in_3')
+                skip_out_0 = gr.Checkbox(label='out_0')
+                skip_out_3 = gr.Checkbox(label='out_3')
+            with gr.Row(variant='compact'):
+                skip_in_1  = gr.Checkbox(label='in_1')
+                skip_in_4  = gr.Checkbox(label='in_4')
+                skip_out_1 = gr.Checkbox(label='out_1')
+                skip_out_4 = gr.Checkbox(label='out_4')
+            with gr.Row(variant='compact'):
+                skip_in_2  = gr.Checkbox(label='in_2')
+                skip_in_5  = gr.Checkbox(label='in_5')
+                skip_out_2 = gr.Checkbox(label='out_2')
+                skip_out_5 = gr.Checkbox(label='out_5')
+            with gr.Row(variant='compact'):
+                skip_mid   = gr.Checkbox(label='mid')
+
+        with gr.Row(variant='compact', visible=DEFAULT_UPSCALE) as tab_ext_upscale:
+            upscale_meth   = gr.Dropdown(label=LABEL_UPSCALE_METH,   value=lambda: DEFAULT_UPSCALE_METH,   choices=CHOICES_UPSCALER)
+            upscale_ratio  = gr.Slider  (label=LABEL_UPSCALE_RATIO,  value=lambda: DEFAULT_UPSCALE_RATIO,  minimum=1.0, maximum=16.0, step=0.1)
+            upscale_width  = gr.Slider  (label=LABEL_UPSCALE_WIDTH,  value=lambda: DEFAULT_UPSCALE_WIDTH,  minimum=0,   maximum=2048, step=8)
+            upscale_height = gr.Slider  (label=LABEL_UPSCALE_HEIGHT, value=lambda: DEFAULT_UPSCALE_HEIGHT, minimum=0,   maximum=2048, step=8)
+
+        with gr.Row(variant='compact', visible=DEFAULT_VIDEO) as tab_ext_video:
+            video_fmt  = gr.Dropdown(label=LABEL_VIDEO_FMT,  value=lambda: DEFAULT_VIDEO_FMT, choices=CHOICES_VIDEO_FMT)
+            video_fps  = gr.Number  (label=LABEL_VIDEO_FPS,  value=lambda: DEFAULT_VIDEO_FPS)
+            video_pad  = gr.Number  (label=LABEL_VIDEO_PAD,  value=lambda: DEFAULT_VIDEO_PAD,  precision=0)
+            video_pick = gr.Text    (label=LABEL_VIDEO_PICK, value=lambda: DEFAULT_VIDEO_PICK, max_lines=1)
+
+        with gr.Row(variant='compact') as tab_ext:
+            ext_video     = gr.Checkbox(label=LABEL_VIDEO,      value=lambda: DEFAULT_VIDEO)
+            ext_upscale   = gr.Checkbox(label=LABEL_UPSCALE,    value=lambda: DEFAULT_UPSCALE)
+            ext_skip_fuse = gr.Checkbox(label=LABEL_SKIP_FUSE,  value=lambda: DEFAULT_SKIP_FUSE)
+            dbg_rife      = gr.Checkbox(label=LABEL_DEBUG_RIFE, value=lambda: DEFAULT_DEBUG_RIFE)
+        
+            ext_video    .change(gr_show, inputs=ext_video,     outputs=tab_ext_video,     show_progress=False)
+            ext_upscale  .change(gr_show, inputs=ext_upscale,   outputs=tab_ext_upscale,   show_progress=False)
+            ext_skip_fuse.change(gr_show, inputs=ext_skip_fuse, outputs=tab_ext_skip_fuse, show_progress=False)
+
+        skip_fuses = [
+            skip_in_0,
+            skip_in_1,
+            skip_in_2,
+            skip_in_3,
+            skip_in_4,
+            skip_in_5,
+            skip_mid,
+            skip_out_0,
+            skip_out_1,
+            skip_out_2,
+            skip_out_3,
+            skip_out_4,
+            skip_out_5,
+        ]
+        return [
+            interp_meth, steps, ctrlnet_ref_dir,
+            upscale_meth, upscale_ratio, upscale_width, upscale_height,
+            video_fmt, video_fps, video_pad, video_pick,
+            ext_video, ext_upscale, ext_skip_fuse, dbg_rife,
+            *skip_fuses,
+        ]
+
+    def run(self, p:Processing, 
+            interp_meth:str, steps:str, ctrlnet_ref_dir:str, 
+            upscale_meth:str, upscale_ratio:float, upscale_width:int, upscale_height:int,
+            video_fmt:str, video_fps:float, video_pad:int, video_pick:str,
+            ext_video:bool, ext_upscale:bool, ext_skip_fuse:bool, dbg_rife:bool,
+            *skip_fuses:bool,
+        ):
+
+        # Prepare ControlNet
+        #self.controlnet_script: ControlNetScript = None
+        self.controlnet_script = None
+        try:
+            for script in p.scripts.alwayson_scripts:
+                if hasattr(script, "latest_network") and script.title().lower() == "controlnet":
+                    script_args: Tuple[ControlNetUnit] = p.script_args[script.args_from:script.args_to]
+                    if not any([u.enabled for u in script_args]): return Processed(p, [], p.seed, f'{CTRLNET_REPO_NAME} not enabled')
+                    self.controlnet_script = script
+                    break
+        except ImportError:
+            return Processed(p, [], p.seed, f'{CTRLNET_REPO_NAME} not installed')
+        except:
+            print_exc()
+        if not self.controlnet_script: return Processed(p, [], p.seed, f'{CTRLNET_REPO_NAME} not loaded')
+
+        # Enum lookup
+        interp_meth: InterpMethod = InterpMethod(interp_meth)
+        video_fmt:   VideoFormat  = VideoFormat (video_fmt)
+
+        # Param check & type convert
+        if ext_video:
+            if video_pad <  0: return Processed(p, [], p.seed, f'video_pad must >= 0, but got {video_pad}')
+            if video_fps <= 0: return Processed(p, [], p.seed, f'video_fps must > 0, but got {video_fps}')
+            try: video_slice = parse_slice(video_pick)
+            except: return Processed(p, [], p.seed, 'syntax error in video_slice')
+        if ext_skip_fuse:
+            global skip_fuse_plan
+            skip_fuse_plan = skip_fuses
+
+        # Prepare ref-images
+        if not ctrlnet_ref_dir: return Processed(p, [], p.seed, f'invalid image folder path: {ctrlnet_ref_dir}')
+        ctrlnet_ref_dir: Path  = Path(ctrlnet_ref_dir)
+        if not ctrlnet_ref_dir.is_dir(): return Processed(p, [], p.seed, f'invalid image folder path: {ctrlnet_ref_dir}(')
+        self.ctrlnet_ref_fps = [fp for fp in list(ctrlnet_ref_dir.iterdir()) if fp.suffix.lower() in ['.jpg', '.jpeg', '.png', '.bmp', '.webp']]
+        n_stages = len(self.ctrlnet_ref_fps)
+        if n_stages == 0: return Processed(p, [], p.seed, f'no images file (*.jpg/*.png/*.bmp/*.webp) found in folder path: {ctrlnet_ref_dir}')
+        if n_stages == 1: return Processed(p, [], p.seed, 'requires at least two images to travel between, but found only 1 :(')
+
+        # Prepare steps (n_interp)
+        try: steps: List[int] = [int(s.strip()) for s in steps.strip().split(',')]
+        except: return Processed(p, [], p.seed, f'cannot parse steps options: {steps}')
+        if   len(steps) == 1: steps = [steps[0]] * (n_stages - 1)
+        elif len(steps) != n_stages - 1: return Processed(p, [], p.seed, f'stage count mismatch: len_steps({len(steps)}) != n_stages({n_stages} - 1))')
+        n_frames = sum(steps) + n_stages
+        if 'show_debug':
+            print('n_stages:', n_stages)
+            print('n_frames:', n_frames)
+            print('steps:', steps)
+        steps.insert(0, -1)     # fixup the first stage
+
+        # Custom saving path
+        travel_path = os.path.join(p.outpath_samples, 'prompt_travel')
+        os.makedirs(travel_path, exist_ok=True)
+        travel_number = get_next_sequence_number(travel_path)
+        self.log_dp = os.path.join(travel_path, f'{travel_number:05}')
+        p.outpath_samples = self.log_dp
+        os.makedirs(self.log_dp, exist_ok=True)
+        self.tmp_dp = Path(self.log_dp) / 'ctrl_cond'   # cache for rife
+        self.tmp_fp = self.tmp_dp / 'tmp.png'           # cache for rife
+
+        # Force Batch Count and Batch Size to 1
+        p.n_iter     = 1
+        p.batch_size = 1
+
+        # Random unified const seed
+        p.seed = get_fixed_seed(p.seed)     # fix it to assure all processes using the same major seed
+        self.subseed = p.subseed            # stash it to allow using random subseed for each process (when -1)
+        if 'show_debug':
+            print('seed:',             p.seed)
+            print('subseed:',          p.subseed)
+            print('subseed_strength:', p.subseed_strength)
+        
+        # Start job
+        state.job_count = n_frames
+
+        # Pack params
+        self.n_stages    = n_stages
+        self.steps       = steps
+        self.interp_meth = interp_meth
+        self.dbg_rife    = dbg_rife
+
+        def upscale_image_callback(params:ImageSaveParams):
+            params.image = upscale_image(params.image, p.width, p.height, upscale_meth, upscale_ratio, upscale_width, upscale_height)
+
+        images: List[PILImage] = []
+        info: str = None
+        try:
+            if ext_upscale: on_before_image_saved(upscale_image_callback)
+
+            self.UnetHook_hook_original = UnetHook.hook
+            UnetHook.hook = hook_hijack
+
+            [c.clear() for c in caches]
+            images, info = self.run_linear(p)
+        except:
+            info = format_exc()
+            print(info)
+        finally:
+            if self.tmp_fp.exists(): os.unlink(self.tmp_fp)
+            [c.clear() for c in caches]
+
+            UnetHook.hook = self.UnetHook_hook_original
+
+            self.controlnet_script.input_image = None
+            if self.controlnet_script.latest_network:
+                self.controlnet_script.latest_network: UnetHook
+                self.controlnet_script.latest_network.restore(p.sd_model.model.diffusion_model)
+                self.controlnet_script.latest_network = None
+
+            if ext_upscale: remove_callbacks_for_function(upscale_image_callback)
+
+            reset_cuda()
+
+        # Save video
+        if ext_video: save_video(images, video_slice, video_pad, video_fps, video_fmt, os.path.join(self.log_dp, f'travel-{travel_number:05}'))
+
+        return Processed(p, images, p.seed, info)
+
+    def run_linear(self, p:Processing) -> RunResults:
+        global from_hint_cond, to_hint_cond, from_control_tensors, to_control_tensors, interp_alpha, interp_ip
+
+        images: List[PILImage] = []
+        info: str = None
+        def process_p(append:bool=True) -> Optional[List[PILImage]]:
+            nonlocal p, images, info
+            proc = process_images(p)
+            if not info: info = proc.info
+            if append: images.extend(proc.images)
+            else: return proc.images
+
+        ''' ↓↓↓ rife interp utils ↓↓↓ '''
+        def save_ctrl_cond(idx:int):
+            self.tmp_dp.mkdir(exist_ok=True)
+            for i, x in enumerate(to_hint_cond):
+                x = x[0]
+                if len(x.shape) == 3:
+                    if   x.shape[0] == 1: x = x.squeeze_(0)         # [C=1, H, W] => [H, W]
+                    elif x.shape[0] == 3: x = x.permute([1, 2, 0])  # [C=3, H, W] => [H, W, C]
+                    else: raise ValueError(f'unknown cond shape: {x.shape}')
+                else:
+                    raise ValueError(f'unknown cond shape: {x.shape}')
+                im = (x.detach().clamp(0.0, 1.0).cpu().numpy() * 255).astype(np.uint8)
+                Image.fromarray(im).save(self.tmp_dp / f'{idx}-{i}.png')
+        def rife_interp(i:int, j:int, k:int, alpha:float) -> Tensor:
+            ''' interp between i-th and j-th cond of the k-th ctrlnet set '''
+            fp0 = self.tmp_dp / f'{i}-{k}.png'
+            fp1 = self.tmp_dp / f'{j}-{k}.png'
+            fpo = self.tmp_dp / f'{i}-{j}-{alpha:.3f}.png' if self.dbg_rife else self.tmp_fp
+            assert run_cmd(f'rife-ncnn-vulkan -m rife-v4 -s {alpha:.3f} -0 "{fp0}" -1 "{fp1}" -o "{fpo}"')
+            x = torch.from_numpy(np.asarray(Image.open(fpo)) / 255.0)
+            if   len(x.shape) == 2: x = x.unsqueeze_(0)             # [H, W] => [C=1, H, W]
+            elif len(x.shape) == 3: x = x.permute([2, 0, 1])        # [H, W, C] => [C, H, W]
+            else: raise ValueError(f'unknown cond shape: {x.shape}')
+            x = x.unsqueeze(dim=0)
+            return x
+        ''' ↑↑↑ rife interp utils ↑↑↑ '''
+
+        ''' ↓↓↓ filename reorder utils ↓↓↓ '''
+        iframe = 0
+        def rename_image_filename(idx:int, param: ImageSaveParams):
+            fn = param.filename
+            stem, suffix = os.path.splitext(os.path.basename(fn))
+            param.filename = os.path.join(os.path.dirname(fn), f'{idx:05d}' + suffix)
+        class on_before_image_saved_wrapper:
+            def __init__(self, callback_fn):
+                self.callback_fn = callback_fn
+            def __enter__(self):
+                on_before_image_saved(self.callback_fn)
+            def __exit__(self, exc_type, exc_value, exc_traceback):
+                remove_callbacks_for_function(self.callback_fn)
+        ''' ↑↑↑ filename reorder utils ↑↑↑ '''
+
+        # Step 1: draw the init image
+        setattr(p, 'init_images', [Image.open(self.ctrlnet_ref_fps[0])])
+        interp_alpha = 0.0
+        with on_before_image_saved_wrapper(partial(rename_image_filename, 0)):
+            process_p()
+            iframe += 1
+        save_ctrl_cond(0)
+
+        # travel through stages
+        for i in range(1, self.n_stages):
+            if state.interrupted: break
+
+            # Setp 3: move to next stage
+            from_hint_cond       = [t for t in to_hint_cond]       ; to_hint_cond      .clear()
+            from_control_tensors = [t for t in to_control_tensors] ; to_control_tensors.clear()
+            setattr(p, 'init_images', [Image.open(self.ctrlnet_ref_fps[i])])
+            interp_alpha = 0.0
+
+            with on_before_image_saved_wrapper(partial(rename_image_filename, iframe + self.steps[i])):
+                cached_images = process_p(append=False)
+            save_ctrl_cond(i)
+
+            # Step 2: draw the interpolated images
+            is_interrupted = False
+            n_inter = self.steps[i] + 1
+            for t in range(1, n_inter):
+                if state.interrupted: is_interrupted = True ; break
+
+                interp_alpha = t / n_inter     # [1/T, 2/T, .. T-1/T]
+
+                mid_hint_cond.clear()
+                device = devices.get_device_for("controlnet")
+                if self.interp_meth == InterpMethod.LINEAR:
+                    for hintA, hintB in zip(from_hint_cond, to_hint_cond):
+                        hintC = weighted_sum(hintA.to(device), hintB.to(device), interp_alpha)
+                        mid_hint_cond.append(hintC)
+                elif self.interp_meth == InterpMethod.RIFE:
+                    dtype = to_hint_cond[0].dtype
+                    for k in range(len(to_hint_cond)):
+                        hintC = rife_interp(i-1, i, k, interp_alpha).to(device, dtype)
+                        mid_hint_cond.append(hintC)
+                else: raise ValueError(f'unknown interp_meth: {self.interp_meth}')
+
+                interp_ip = 0
+                with on_before_image_saved_wrapper(partial(rename_image_filename, iframe)):
+                    process_p()
+                    iframe += 1
+
+            # adjust order
+            images.extend(cached_images)
+            iframe += 1
+
+            if is_interrupted: break
+
+        return images, info

scripts/prompt_travel.py

@@ -0,0 +1,818 @@
+# This extension works with [https://github.com/AUTOMATIC1111/stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui)
+# version: v1.4.0
+
+LOG_PREFIX = '[Prompt-Travel]'
+
+import os
+from pathlib import Path
+from PIL.Image import Image as PILImage
+from enum import Enum
+from dataclasses import dataclass
+from functools import partial
+from typing import List, Tuple, Callable, Any, Optional, Generic, TypeVar
+from traceback import print_exc, format_exc
+
+import gradio as gr
+import numpy as np
+import torch
+from torch import Tensor
+import torch.nn.functional as F
+try:
+    from moviepy.video.io.ImageSequenceClip import ImageSequenceClip
+    from moviepy.editor import concatenate_videoclips, ImageClip
+except ImportError:
+    print(f'{LOG_PREFIX} package moviepy not installed, will not be able to generate video')
+
+import modules.scripts as scripts
+from modules.script_callbacks import on_before_image_saved, ImageSaveParams, on_cfg_denoiser, CFGDenoiserParams, remove_callbacks_for_function
+from modules.ui import gr_show
+from modules.shared import state, opts, sd_upscalers
+from modules.processing import process_images, get_fixed_seed
+from modules.processing import Processed, StableDiffusionProcessing as Processing, StableDiffusionProcessingTxt2Img as ProcessingTxt2Img, StableDiffusionProcessingImg2Img as ProcessingImg2Img
+from modules.images import resize_image
+from modules.sd_samplers_common import single_sample_to_image
+
+class Mode(Enum):
+    LINEAR  = 'linear'
+    REPLACE = 'replace'
+
+class LerpMethod(Enum):
+    LERP  = 'lerp'
+    SLERP = 'slerp'
+
+class ModeReplaceDim(Enum):
+    TOKEN   = 'token'
+    CHANNEL = 'channel'
+    RANDOM  = 'random'
+
+class ModeReplaceOrder(Enum):
+    SIMILAR   = 'similar'
+    DIFFERENT = 'different'
+    RANDOM    = 'random'
+
+class Gensis(Enum):
+    FIXED      = 'fixed'
+    SUCCESSIVE = 'successive'
+    EMBRYO     = 'embryo'
+
+class VideoFormat(Enum):
+    MP4  = 'mp4'
+    GIF  = 'gif'
+    WEBM = 'webm'
+
+if 'typing':
+    T = TypeVar('T')
+    @dataclass
+    class Ref(Generic[T]): value: T = None
+
+    TensorRef = Ref[Tensor]
+    StrRef = Ref[str]
+    PILImages = List[PILImage]
+    RunResults = Tuple[PILImages, str]
+
+if 'consts':
+    __ = lambda key, value=None: opts.data.get(f'customscript/prompt_travel.py/txt2img/{key}/value', value)
+
+    LABEL_MODE              = 'Travel mode'
+    LABEL_STEPS             = 'Travel steps between stages'
+    LABEL_GENESIS           = 'Frame genesis'
+    LABEL_DENOISE_W         = 'Denoise strength'
+    LABEL_EMBRYO_STEP       = 'Denoise steps for embryo'
+    LABEL_LERP_METH         = 'Linear interp method'
+    LABEL_REPLACE_DIM       = 'Replace dimension'
+    LABEL_REPLACE_ORDER     = 'Replace order'
+    LABEL_VIDEO             = 'Ext. export video'
+    LABEL_VIDEO_FPS         = 'Video FPS'
+    LABEL_VIDEO_FMT         = 'Video file format'
+    LABEL_VIDEO_PAD         = 'Pad begin/end frames'
+    LABEL_VIDEO_PICK        = 'Pick frame by slice'
+    LABEL_UPSCALE           = 'Ext. upscale'
+    LABEL_UPSCALE_METH      = 'Upscaler'
+    LABEL_UPSCALE_RATIO     = 'Upscale ratio'
+    LABEL_UPSCALE_WIDTH     = 'Upscale width'
+    LABEL_UPSCALE_HEIGHT    = 'Upscale height'
+    LABEL_DEPTH             = 'Ext. depth-image-io (for depth2img models)'
+    LABEL_DEPTH_IMG         = 'Depth image file'
+
+    DEFAULT_MODE            = __(LABEL_MODE, Mode.LINEAR.value)
+    DEFAULT_STEPS           = __(LABEL_STEPS, 30)
+    DEFAULT_GENESIS         = __(LABEL_GENESIS, Gensis.FIXED.value)
+    DEFAULT_DENOISE_W       = __(LABEL_DENOISE_W, 1.0)
+    DEFAULT_EMBRYO_STEP     = __(LABEL_EMBRYO_STEP, 8)
+    DEFAULT_LERP_METH       = __(LABEL_LERP_METH, LerpMethod.LERP.value)
+    DEFAULT_REPLACE_DIM     = __(LABEL_REPLACE_DIM, ModeReplaceDim.TOKEN.value)
+    DEFAULT_REPLACE_ORDER   = __(LABEL_REPLACE_ORDER, ModeReplaceOrder.RANDOM.value)
+    DEFAULT_UPSCALE         = __(LABEL_UPSCALE, False)
+    DEFAULT_UPSCALE_METH    = __(LABEL_UPSCALE_METH, 'Lanczos')
+    DEFAULT_UPSCALE_RATIO   = __(LABEL_UPSCALE_RATIO, 2.0)
+    DEFAULT_UPSCALE_WIDTH   = __(LABEL_UPSCALE_WIDTH, 0)
+    DEFAULT_UPSCALE_HEIGHT  = __(LABEL_UPSCALE_HEIGHT, 0)
+    DEFAULT_VIDEO           = __(LABEL_VIDEO, True)
+    DEFAULT_VIDEO_FPS       = __(LABEL_VIDEO_FPS, 10)
+    DEFAULT_VIDEO_FMT       = __(LABEL_VIDEO_FMT, VideoFormat.MP4.value)
+    DEFAULT_VIDEO_PAD       = __(LABEL_VIDEO_PAD, 0)
+    DEFAULT_VIDEO_PICK      = __(LABEL_VIDEO_PICK, '')
+    DEFAULT_DEPTH           = __(LABEL_DEPTH, False)
+
+    CHOICES_MODE            = [x.value for x in Mode]
+    CHOICES_LERP_METH       = [x.value for x in LerpMethod]
+    CHOICES_GENESIS         = [x.value for x in Gensis]
+    CHOICES_REPLACE_DIM     = [x.value for x in ModeReplaceDim]
+    CHOICES_REPLACE_ORDER   = [x.value for x in ModeReplaceOrder]
+    CHOICES_UPSCALER        = [x.name for x in sd_upscalers]
+    CHOICES_VIDEO_FMT       = [x.value for x in VideoFormat]
+
+    EPS = 1e-6
+
+
+def cond_align(condA:Tensor, condB:Tensor) -> Tuple[Tensor, Tensor]:
+    d = condA.shape[0] - condB.shape[0]
+    if   d < 0: condA = F.pad(condA, (0, 0, 0, -d))
+    elif d > 0: condB = F.pad(condB, (0, 0, 0,  d))
+    return condA, condB
+
+def wrap_cond_align(fn:Callable[..., Tensor]):
+    def wrapper(condA:Tensor, condB:Tensor, *args, **kwargs) -> Tensor:
+        condA, condB = cond_align(condA, condB)
+        return fn(condA, condB, *args, **kwargs)
+    return wrapper
+
+@wrap_cond_align
+def weighted_sum(condA:Tensor, condB:Tensor, alpha:float) -> Tensor:
+    ''' linear interpolate on latent space of condition '''
+
+    return (1 - alpha) * condA + (alpha) * condB
+
+@wrap_cond_align
+def geometric_slerp(condA:Tensor, condB:Tensor, alpha:float) -> Tensor:
+    ''' spherical linear interpolation on latent space of condition, ref: https://en.wikipedia.org/wiki/Slerp '''
+
+    A_n = condA / torch.norm(condA, dim=-1, keepdim=True)   # [T=77, D=768]
+    B_n = condB / torch.norm(condB, dim=-1, keepdim=True)
+
+    dot = (A_n * B_n).sum(dim=-1, keepdim=True)     # [T=77, D=1]
+    omega = torch.acos(dot)                         # [T=77, D=1]
+    so = torch.sin(omega)                           # [T=77, D=1]
+
+    slerp = (torch.sin((1 - alpha) * omega) / so) * condA + (torch.sin(alpha * omega) / so) * condB
+
+    mask: Tensor = dot > 0.9995                     # [T=77, D=1]
+    if not mask.any():
+        return slerp
+    else:
+        lerp = (1 - alpha) * condA + (alpha) * condB
+        return torch.where(mask, lerp, slerp)       # use simple lerp when angle very close to avoid NaN
+
+@wrap_cond_align
+def replace_until_match(condA:Tensor, condB:Tensor, count:int, dist:Tensor, order:str=ModeReplaceOrder.RANDOM) -> Tensor:
+    ''' value substite on condition tensor; will inplace modify `dist` '''
+
+    def index_tensor_to_tuple(index:Tensor) -> Tuple[Tensor, ...]:
+        return tuple([index[..., i] for i in range(index.shape[-1])])       # tuple([nDiff], ...)
+
+    # mask: [T=77, D=768], [T=77] or [D=768]
+    mask = dist > EPS
+    # idx_diff: [nDiff, nDim=2] or [nDiff, nDim=1]
+    idx_diff = torch.nonzero(mask)
+    n_diff = len(idx_diff)
+
+    if order == ModeReplaceOrder.RANDOM:
+        sel = np.random.choice(range(n_diff), size=count, replace=False) if n_diff > count else slice(None)
+    else:
+        val_diff = dist[index_tensor_to_tuple(idx_diff)]    # [nDiff]
+
+        if order == ModeReplaceOrder.SIMILAR:
+            sorted_index = val_diff.argsort()
+        elif order == ModeReplaceOrder.DIFFERENT:
+            sorted_index = val_diff.argsort(descending=True)
+        else: raise ValueError(f'unknown replace_order: {order}')
+
+        sel = sorted_index[:count]
+
+    idx_diff_sel = idx_diff[sel, ...]       # [cnt] => [cnt, nDim]
+    idx_diff_sel_tp = index_tensor_to_tuple(idx_diff_sel)
+    dist[idx_diff_sel_tp] = 0.0
+    mask[idx_diff_sel_tp] = False
+
+    if mask.shape != condA.shape:   # cond.shape = [T=77, D=768]
+        mask_len = mask.shape[0]
+        if   mask_len == condA.shape[0]: mask = mask.unsqueeze(1)
+        elif mask_len == condA.shape[1]: mask = mask.unsqueeze(0)
+        else: raise ValueError(f'unknown mask.shape: {mask.shape}')
+        mask = mask.expand_as(condA)
+
+    return mask * condA + ~mask * condB
+
+
+def get_next_sequence_number(path:str) -> int:
+    """ Determines and returns the next sequence number to use when saving an image in the specified directory. The sequence starts at 0. """
+    result = -1
+    dir = Path(path)
+    for file in dir.iterdir():
+        if not file.is_dir(): continue
+        try:
+            num = int(file.name)
+            if num > result: result = num
+        except ValueError:
+            pass
+    return result + 1
+
+def update_img2img_p(p:Processing, imgs:PILImages, denoising_strength:float=0.75) -> ProcessingImg2Img:
+    if isinstance(p, ProcessingImg2Img):
+        p.init_images = imgs
+        p.denoising_strength = denoising_strength
+        return p
+
+    if isinstance(p, ProcessingTxt2Img):
+        KNOWN_KEYS = [      # see `StableDiffusionProcessing.__init__()`
+            'sd_model',
+            'outpath_samples',
+            'outpath_grids',
+            'prompt',
+            'styles',
+            'seed',
+            'subseed',
+            'subseed_strength',
+            'seed_resize_from_h',
+            'seed_resize_from_w',
+            'seed_enable_extras',
+            'sampler_name',
+            'batch_size',
+            'n_iter',
+            'steps',
+            'cfg_scale',
+            'width',
+            'height',
+            'restore_faces',
+            'tiling',
+            'do_not_save_samples',
+            'do_not_save_grid',
+            'extra_generation_params',
+            'overlay_images',
+            'negative_prompt',
+            'eta',
+            'do_not_reload_embeddings',
+            #'denoising_strength',
+            'ddim_discretize',
+            's_min_uncond',
+            's_churn',
+            's_tmax',
+            's_tmin',
+            's_noise',
+            'override_settings',
+            'override_settings_restore_afterwards',
+            'sampler_index',
+            'script_args',
+        ]
+        kwargs = { k: getattr(p, k) for k in dir(p) if k in KNOWN_KEYS }    # inherit params
+        return ProcessingImg2Img(
+            init_images=imgs,
+            denoising_strength=denoising_strength,
+            **kwargs,
+        )
+
+def parse_slice(picker:str) -> Optional[slice]:
+    if not picker.strip(): return None
+    
+    to_int = lambda s: None if not s else int(s)
+    segs = [to_int(x.strip()) for x in picker.strip().split(':')]
+    
+    start, stop, step = None, None, None
+    if   len(segs) == 1:        stop,      = segs
+    elif len(segs) == 2: start, stop       = segs
+    elif len(segs) == 3: start, stop, step = segs
+    else: raise ValueError
+    
+    return slice(start, stop, step)
+
+def parse_resolution(width:int, height:int, upscale_ratio:float, upscale_width:int, upscale_height:int) -> Tuple[bool, Tuple[int, int]]:
+    if upscale_width == upscale_height == 0:
+        if upscale_ratio == 1.0:
+            return False, (width, height)
+        else:
+            return True, (round(width * upscale_ratio), round(height * upscale_ratio))
+    else:
+        if upscale_width  == 0: upscale_width  = round(width  * upscale_height / height)
+        if upscale_height == 0: upscale_height = round(height * upscale_width  / width)
+        return (width != upscale_width and height != upscale_height), (upscale_width, upscale_height)
+
+
+def upscale_image(img:PILImage, width:int, height:int, upscale_meth:str, upscale_ratio:float, upscale_width:int, upscale_height:int) -> PILImage:
+    if upscale_meth == 'None': return img
+    need_upscale, (tgt_w, tgt_h) = parse_resolution(width, height, upscale_ratio, upscale_width, upscale_height)
+    if need_upscale:
+        if 'show_debug': print(f'>> upscale: ({width}, {height}) => ({tgt_w}, {tgt_h})')
+
+        if max(tgt_w / width, tgt_h / height) > 4:      # must split into two rounds for NN model capatibility
+            hf_w, hf_h = round(width * 4), round(height * 4)
+            img = resize_image(0, img, hf_w, hf_h, upscaler_name=upscale_meth)
+        img = resize_image(0, img, tgt_w, tgt_h, upscaler_name=upscale_meth)
+    return img
+
+def save_video(imgs:PILImages, video_slice:slice, video_pad:int, video_fps:float, video_fmt:VideoFormat, fbase:str):
+    if len(imgs) <= 1 or 'ImageSequenceClip' not in globals(): return
+
+    try:
+        # arrange frames
+        if video_slice:   imgs = imgs[video_slice]
+        if video_pad > 0: imgs = [imgs[0]] * video_pad + imgs + [imgs[-1]] * video_pad
+
+        # export video
+        seq: List[np.ndarray] = [np.asarray(img) for img in imgs]
+        try:
+            clip = ImageSequenceClip(seq, fps=video_fps)
+        except:     # images may have different size (small probability due to upscaler)
+            clip = concatenate_videoclips([ImageClip(img, duration=1/video_fps) for img in seq], method='compose')
+            clip.fps = video_fps
+        if   video_fmt == VideoFormat.MP4:  clip.write_videofile(fbase + '.mp4',  verbose=False, audio=False)
+        elif video_fmt == VideoFormat.WEBM: clip.write_videofile(fbase + '.webm', verbose=False, audio=False)
+        elif video_fmt == VideoFormat.GIF:  clip.write_gif      (fbase + '.gif',  loop=True)
+    except: print_exc()
+
+
+class on_cfg_denoiser_wrapper:
+    def __init__(self, callback_fn:Callable):
+        self.callback_fn = callback_fn
+    def __enter__(self):
+        on_cfg_denoiser(self.callback_fn)
+    def __exit__(self, exc_type, exc_value, exc_traceback):
+        remove_callbacks_for_function(self.callback_fn)
+
+class p_steps_overrider:
+    def __init__(self, p:Processing, steps:int=1):
+        self.p = p
+        self.steps = steps
+        self.steps_saved = self.p.steps
+    def __enter__(self):
+        self.p.steps = self.steps
+    def __exit__(self, exc_type, exc_value, exc_traceback):
+        self.p.steps = self.steps_saved
+
+class p_save_samples_overrider:
+    def __init__(self, p:Processing, save:bool=True):
+        self.p = p
+        self.save = save
+        self.do_not_save_samples_saved = self.p.do_not_save_samples
+    def __enter__(self):
+        self.p.do_not_save_samples = not self.save
+    def __exit__(self, exc_type, exc_value, exc_traceback):
+        self.p.do_not_save_samples = self.do_not_save_samples_saved
+
+def get_cond_callback(refs:List[TensorRef], params:CFGDenoiserParams):
+    if params.sampling_step > 0: return
+    values = [
+        params.text_cond,        # [B=1, L= 77, D=768]
+        params.text_uncond,      # [B=1, L=231, D=768]
+    ]
+    for i, ref in enumerate(refs):
+        ref.value = values[i]
+
+def set_cond_callback(refs:List[TensorRef], params:CFGDenoiserParams):
+    values = [
+        params.text_cond,        # [B=1, L= 77, D=768]
+        params.text_uncond,      # [B=1, L=231, D=768]
+    ]
+    for i, ref in enumerate(refs):
+        values[i].data = ref.value
+
+def get_latent_callback(ref:TensorRef, embryo_step:int, params:CFGDenoiserParams):
+    if params.sampling_step != embryo_step: return
+    ref.value = params.x
+
+def set_latent_callback(ref:TensorRef, embryo_step:int, params:CFGDenoiserParams):
+    if params.sampling_step != embryo_step: return
+    params.x.data = ref.value
+
+
+def switch_to_stage_binding_(self:'Script', i:int):
+    if 'show_debug':
+        print(f'[stage {i+1}/{self.n_stages}]')
+        print(f'  pos prompt: {self.pos_prompts[i]}')
+        if hasattr(self, 'neg_prompts'):
+            print(f'  neg prompt: {self.neg_prompts[i]}')
+    self.p.prompt = self.pos_prompts[i]
+    if hasattr(self, 'neg_prompts'):
+        self.p.negative_prompt = self.neg_prompts[i]
+    self.p.subseed = self.subseed
+
+def process_p_binding_(self:'Script', append:bool=True, save:bool=True) -> PILImages:
+    assert hasattr(self, 'images') and hasattr(self, 'info'), 'unknown logic, "images" and "info" not initialized'
+    with p_save_samples_overrider(self.p, save):
+        proc = process_images(self.p)
+    if save:
+        if not self.info.value: self.info.value = proc.info
+        if append: self.images.extend(proc.images)
+        if self.genesis == Gensis.SUCCESSIVE:
+            self.p = update_img2img_p(self.p, self.images[-1:], self.denoise_w)
+        return proc.images
+
+
+class Script(scripts.Script):
+
+    def title(self):
+        return 'Prompt Travel'
+
+    def describe(self):
+        return 'Travel from one prompt to another in the text encoder latent space.'
+
+    def show(self, is_img2img):
+        return True
+
+    def ui(self, is_img2img):
+        with gr.Row(variant='compact') as tab_mode:
+            mode          = gr.Radio   (label=LABEL_MODE,          value=lambda: DEFAULT_MODE,          choices=CHOICES_MODE)
+            lerp_meth     = gr.Dropdown(label=LABEL_LERP_METH,     value=lambda: DEFAULT_LERP_METH,     choices=CHOICES_LERP_METH)
+            replace_dim   = gr.Dropdown(label=LABEL_REPLACE_DIM,   value=lambda: DEFAULT_REPLACE_DIM,   choices=CHOICES_REPLACE_DIM,   visible=False)
+            replace_order = gr.Dropdown(label=LABEL_REPLACE_ORDER, value=lambda: DEFAULT_REPLACE_ORDER, choices=CHOICES_REPLACE_ORDER, visible=False)
+
+            def switch_mode(mode:str):
+                show_meth = Mode(mode) == Mode.LINEAR
+                show_repl = Mode(mode) == Mode.REPLACE
+                return [gr_show(x) for x in [show_meth, show_repl, show_repl]]
+            mode.change(switch_mode, inputs=[mode], outputs=[lerp_meth, replace_dim, replace_order], show_progress=False)
+
+        with gr.Row(variant='compact') as tab_param:
+            steps       = gr.Text    (label=LABEL_STEPS,       value=lambda: DEFAULT_STEPS, max_lines=1)
+            genesis     = gr.Dropdown(label=LABEL_GENESIS,     value=lambda: DEFAULT_GENESIS, choices=CHOICES_GENESIS)
+            denoise_w   = gr.Slider  (label=LABEL_DENOISE_W,   value=lambda: DEFAULT_DENOISE_W, minimum=0.0, maximum=1.0, visible=False)
+            embryo_step = gr.Text    (label=LABEL_EMBRYO_STEP, value=lambda: DEFAULT_EMBRYO_STEP, max_lines=1, visible=False)
+
+            def switch_genesis(genesis:str):
+                show_dw = Gensis(genesis) == Gensis.SUCCESSIVE    # show 'denoise_w' for 'successive'
+                show_es = Gensis(genesis) == Gensis.EMBRYO        # show 'embryo_step' for 'embryo'
+                return [gr_show(x) for x in [show_dw, show_es]]
+            genesis.change(switch_genesis, inputs=[genesis], outputs=[denoise_w, embryo_step], show_progress=False)
+
+        with gr.Row(variant='compact', visible=DEFAULT_DEPTH) as tab_ext_depth:
+            depth_img = gr.Image(label=LABEL_DEPTH_IMG, source='upload', type='pil', image_mode=None)
+
+        with gr.Row(variant='compact', visible=DEFAULT_UPSCALE) as tab_ext_upscale:
+            upscale_meth   = gr.Dropdown(label=LABEL_UPSCALE_METH,   value=lambda: DEFAULT_UPSCALE_METH,   choices=CHOICES_UPSCALER)
+            upscale_ratio  = gr.Slider  (label=LABEL_UPSCALE_RATIO,  value=lambda: DEFAULT_UPSCALE_RATIO,  minimum=1.0, maximum=16.0, step=0.1)
+            upscale_width  = gr.Slider  (label=LABEL_UPSCALE_WIDTH,  value=lambda: DEFAULT_UPSCALE_WIDTH,  minimum=0, maximum=2048, step=8)
+            upscale_height = gr.Slider  (label=LABEL_UPSCALE_HEIGHT, value=lambda: DEFAULT_UPSCALE_HEIGHT, minimum=0, maximum=2048, step=8)
+
+        with gr.Row(variant='compact', visible=DEFAULT_VIDEO) as tab_ext_video:
+            video_fmt  = gr.Dropdown(label=LABEL_VIDEO_FMT,  value=lambda: DEFAULT_VIDEO_FMT, choices=CHOICES_VIDEO_FMT)
+            video_fps  = gr.Number  (label=LABEL_VIDEO_FPS,  value=lambda: DEFAULT_VIDEO_FPS)
+            video_pad  = gr.Number  (label=LABEL_VIDEO_PAD,  value=lambda: DEFAULT_VIDEO_PAD,  precision=0)
+            video_pick = gr.Text    (label=LABEL_VIDEO_PICK, value=lambda: DEFAULT_VIDEO_PICK, max_lines=1)
+
+        with gr.Row(variant='compact') as tab_ext:
+            ext_video   = gr.Checkbox(label=LABEL_VIDEO,   value=lambda: DEFAULT_VIDEO) 
+            ext_upscale = gr.Checkbox(label=LABEL_UPSCALE, value=lambda: DEFAULT_UPSCALE) 
+            ext_depth   = gr.Checkbox(label=LABEL_DEPTH,   value=lambda: DEFAULT_DEPTH)
+        
+            ext_video  .change(gr_show, inputs=ext_video,   outputs=tab_ext_video,   show_progress=False)
+            ext_upscale.change(gr_show, inputs=ext_upscale, outputs=tab_ext_upscale, show_progress=False)
+            ext_depth  .change(gr_show, inputs=ext_depth,   outputs=tab_ext_depth,   show_progress=False)
+
+        return [
+            mode, lerp_meth, replace_dim, replace_order,
+            steps, genesis, denoise_w, embryo_step,
+            depth_img,
+            upscale_meth, upscale_ratio, upscale_width, upscale_height,
+            video_fmt, video_fps, video_pad, video_pick,
+            ext_video, ext_upscale, ext_depth,
+        ]
+
+    def run(self, p:Processing, 
+            mode:str, lerp_meth:str, replace_dim:str, replace_order:str,
+            steps:str, genesis:str, denoise_w:float, embryo_step:str,
+            depth_img:PILImage,
+            upscale_meth:str, upscale_ratio:float, upscale_width:int, upscale_height:int,
+            video_fmt:str, video_fps:float, video_pad:int, video_pick:str,
+            ext_video:bool, ext_upscale:bool, ext_depth:bool,
+        ):
+        
+        # enum lookup
+        mode: Mode                      = Mode(mode)
+        lerp_meth: LerpMethod           = LerpMethod(lerp_meth)
+        replace_dim: ModeReplaceDim     = ModeReplaceDim(replace_dim)
+        replace_order: ModeReplaceOrder = ModeReplaceOrder(replace_order)
+        genesis: Gensis                 = Gensis(genesis)
+        video_fmt: VideoFormat          = VideoFormat(video_fmt)
+
+        # Param check & type convert
+        if ext_video:
+            if video_pad <  0: return Processed(p, [], p.seed, f'video_pad must >= 0, but got {video_pad}')
+            if video_fps <= 0: return Processed(p, [], p.seed, f'video_fps must > 0, but got {video_fps}')
+            try: video_slice = parse_slice(video_pick)
+            except: return Processed(p, [], p.seed, 'syntax error in video_slice')
+        if genesis == Gensis.EMBRYO:
+            try: x = float(embryo_step)
+            except: return Processed(p, [], p.seed, f'embryo_step is not a number: {embryo_step}')
+            if x <= 0: return Processed(p, [], p.seed, f'embryo_step must > 0, but got {embryo_step}')
+            embryo_step: int = round(x * p.steps if x < 1.0 else x) ; del x
+
+        # Prepare prompts & steps
+        prompt_pos = p.prompt.strip()
+        if not prompt_pos: return Processed(p, [], p.seed, 'positive prompt should not be empty :(')
+        pos_prompts = [p.strip() for p in prompt_pos.split('\n') if p.strip()]
+        if len(pos_prompts) == 1: return Processed(p, [], p.seed, 'should specify at least two lines of prompt to travel between :(')
+        if genesis == Gensis.EMBRYO and len(pos_prompts) > 2: return Processed(p, [], p.seed, 'processing with "embryo" genesis takes exactly two lines of prompt :(')
+        prompt_neg = p.negative_prompt.strip()
+        neg_prompts = [p.strip() for p in prompt_neg.split('\n') if p.strip()]
+        if len(neg_prompts) == 0: neg_prompts = ['']
+        n_stages = max(len(pos_prompts), len(neg_prompts))
+        while len(pos_prompts) < n_stages: pos_prompts.append(pos_prompts[-1])
+        while len(neg_prompts) < n_stages: neg_prompts.append(neg_prompts[-1])
+
+        try: steps: List[int] = [int(s.strip()) for s in steps.strip().split(',')]
+        except: return Processed(p, [], p.seed, f'cannot parse steps option: {steps}')
+        if len(steps) == 1:
+            steps = [steps[0]] * (n_stages - 1)
+        elif len(steps) != n_stages - 1:
+            return Processed(p, [], p.seed, f'stage count mismatch: you have {n_stages} prompt stages, but specified {len(steps)} steps; should assure len(steps) == len(stages) - 1')
+        n_frames = sum(steps) + n_stages
+        if 'show_debug':
+            print('n_stages:', n_stages)
+            print('n_frames:', n_frames)
+            print('steps:', steps)
+        steps.insert(0, -1)     # fixup the first stage
+
+        # Custom saving path
+        travel_path = os.path.join(p.outpath_samples, 'prompt_travel')
+        os.makedirs(travel_path, exist_ok=True)
+        travel_number = get_next_sequence_number(travel_path)
+        self.log_dp = os.path.join(travel_path, f'{travel_number:05}')
+        p.outpath_samples = self.log_dp
+        os.makedirs(self.log_dp, exist_ok=True)
+        #self.log_fp = os.path.join(self.log_dp, 'log.txt')
+
+        # Force batch count and batch size to 1
+        p.n_iter     = 1
+        p.batch_size = 1
+
+        # Random unified const seed
+        p.seed = get_fixed_seed(p.seed)     # fix it to assure all processes using the same major seed
+        self.subseed = p.subseed            # stash it to allow using random subseed for each process (when -1)
+        if 'show_debug':
+            print('seed:',             p.seed)
+            print('subseed:',          p.subseed)
+            print('subseed_strength:', p.subseed_strength)
+
+        # Start job
+        state.job_count = n_frames
+
+        # Pack parameters
+        self.pos_prompts   = pos_prompts
+        self.neg_prompts   = neg_prompts
+        self.steps         = steps
+        self.genesis       = genesis
+        self.denoise_w     = denoise_w
+        self.embryo_step   = embryo_step
+        self.lerp_meth     = lerp_meth
+        self.replace_dim   = replace_dim
+        self.replace_order = replace_order
+        self.n_stages      = n_stages
+        self.n_frames      = n_frames
+
+        def upscale_image_callback(params:ImageSaveParams):
+            params.image = upscale_image(params.image, p.width, p.height, upscale_meth, upscale_ratio, upscale_width, upscale_height)
+
+        # Dispatch
+        self.p: Processing = p
+        self.images: PILImages = []
+        self.info: StrRef = Ref()
+        try:
+            if ext_upscale: on_before_image_saved(upscale_image_callback)
+            if ext_depth: self.ext_depth_preprocess(p, depth_img)
+
+            runner = getattr(self, f'run_{mode.value}')
+            if not runner: return Processed(p, [], p.seed, f'no runner found for mode: {mode.value}')
+            runner()
+        except:
+            e = format_exc()
+            print(e)
+            self.info.value = e
+        finally:
+            if ext_depth: self.ext_depth_postprocess(p, depth_img)
+            if ext_upscale: remove_callbacks_for_function(upscale_image_callback)
+
+        # Save video
+        if ext_video: save_video(self.images, video_slice, video_pad, video_fps, video_fmt, os.path.join(self.log_dp, f'travel-{travel_number:05}'))
+
+        return Processed(p, self.images, p.seed, self.info.value)
+
+    def run_linear(self):
+        # dispatch for special case
+        if self.genesis == Gensis.EMBRYO: return self.run_linear_embryo()
+
+        lerp_fn = weighted_sum if self.lerp_meth == LerpMethod.LERP else geometric_slerp
+
+        if 'auxiliary':
+            switch_to_stage = partial(switch_to_stage_binding_, self)
+            process_p = partial(process_p_binding_, self)
+
+            from_pos_hidden:  TensorRef = Ref()
+            from_neg_hidden:  TensorRef = Ref()
+            to_pos_hidden:    TensorRef = Ref()
+            to_neg_hidden:    TensorRef = Ref()
+            inter_pos_hidden: TensorRef = Ref()
+            inter_neg_hidden: TensorRef = Ref()
+
+        # Step 1: draw the init image
+        switch_to_stage(0)
+        with on_cfg_denoiser_wrapper(partial(get_cond_callback, [from_pos_hidden, from_neg_hidden])):
+            process_p()
+
+        # travel through stages
+        for i in range(1, self.n_stages):
+            if state.interrupted: break
+
+            state.job = f'{i}/{self.n_frames}'
+            state.job_no = i + 1
+
+            # only change target prompts
+            switch_to_stage(i)
+            with on_cfg_denoiser_wrapper(partial(get_cond_callback, [to_pos_hidden, to_neg_hidden])):
+                if self.genesis == Gensis.FIXED:
+                    imgs = process_p(append=False)      # stash it to make order right
+                elif self.genesis == Gensis.SUCCESSIVE:
+                    with p_steps_overrider(self.p, steps=1):    # ignore final image, only need cond
+                        process_p(save=False, append=False)
+                else: raise ValueError(f'invalid genesis: {self.genesis.value}')
+
+            # Step 2: draw the interpolated images
+            is_break_iter = False
+            n_inter = self.steps[i]
+            for t in range(1, n_inter + (1 if self.genesis == Gensis.SUCCESSIVE else 0)):
+                if state.interrupted: is_break_iter = True ; break
+
+                alpha = t / n_inter     # [1/T, 2/T, .. T-1/T] (+ [T/T])?
+                inter_pos_hidden.value = lerp_fn(from_pos_hidden.value, to_pos_hidden.value, alpha)
+                inter_neg_hidden.value = lerp_fn(from_neg_hidden.value, to_neg_hidden.value, alpha)
+                with on_cfg_denoiser_wrapper(partial(set_cond_callback, [inter_pos_hidden, inter_neg_hidden])):
+                    process_p()
+
+            if is_break_iter: break
+
+            # Step 3: append the final stage
+            if self.genesis != Gensis.SUCCESSIVE: self.images.extend(imgs)
+            # move to next stage
+            from_pos_hidden.value, from_neg_hidden.value = to_pos_hidden.value, to_neg_hidden.value
+            inter_pos_hidden.value, inter_neg_hidden.value = None, None
+
+    def run_linear_embryo(self):
+        ''' NOTE: this procedure has special logic, we separate it from run_linear() so far '''
+
+        lerp_fn  = weighted_sum if self.lerp_meth == LerpMethod.LERP else geometric_slerp
+        n_frames = self.steps[1] + 2
+
+        if 'auxiliary':
+            switch_to_stage = partial(switch_to_stage_binding_, self)
+            process_p = partial(process_p_binding_, self)
+
+            from_pos_hidden:  TensorRef = Ref()
+            to_pos_hidden:    TensorRef = Ref()
+            inter_pos_hidden: TensorRef = Ref()
+            embryo:           TensorRef = Ref()     # latent image, the common half-denoised prototype of all frames
+
+        # Step 1: get starting & ending condition
+        switch_to_stage(0)
+        with on_cfg_denoiser_wrapper(partial(get_cond_callback, [from_pos_hidden])):
+            with p_steps_overrider(self.p, steps=1):
+                process_p(save=False)
+        switch_to_stage(1)
+        with on_cfg_denoiser_wrapper(partial(get_cond_callback, [to_pos_hidden])):
+            with p_steps_overrider(self.p, steps=1):
+                process_p(save=False)
+
+        # Step 2: get the condition middle-point as embryo then hatch it halfway
+        inter_pos_hidden.value = lerp_fn(from_pos_hidden.value, to_pos_hidden.value, 0.5)
+        with on_cfg_denoiser_wrapper(partial(set_cond_callback, [inter_pos_hidden])):
+            with on_cfg_denoiser_wrapper(partial(get_latent_callback, embryo, self.embryo_step)):
+                process_p(save=False)
+        try:
+            img: PILImage = single_sample_to_image(embryo.value[0], approximation=-1)     # the data is duplicated, just get first item
+            img.save(os.path.join(self.log_dp, 'embryo.png'))
+        except: pass
+
+        # Step 3: derive the embryo towards each interpolated condition
+        for t in range(0, n_frames+1):
+            if state.interrupted: break
+
+            alpha = t / n_frames     # [0, 1/T, 2/T, .. T-1/T, 1]
+            inter_pos_hidden.value = lerp_fn(from_pos_hidden.value, to_pos_hidden.value, alpha)
+            with on_cfg_denoiser_wrapper(partial(set_cond_callback, [inter_pos_hidden])):
+                with on_cfg_denoiser_wrapper(partial(set_latent_callback, embryo, self.embryo_step)):
+                    process_p()
+
+    def run_replace(self):
+        ''' yet another replace method, but do replacing on the condition tensor by token dim or channel dim '''
+
+        if self.genesis == Gensis.EMBRYO: raise NotImplementedError(f'genesis {self.genesis.value!r} is only supported in linear mode currently :(')
+
+        if 'auxiliary':
+            switch_to_stage = partial(switch_to_stage_binding_, self)
+            process_p = partial(process_p_binding_, self)
+
+            from_pos_hidden:  TensorRef = Ref()
+            to_pos_hidden:    TensorRef = Ref()
+            inter_pos_hidden: TensorRef = Ref()
+
+        # Step 1: draw the init image
+        switch_to_stage(0)
+        with on_cfg_denoiser_wrapper(partial(get_cond_callback, [from_pos_hidden])):
+            process_p()
+        
+        # travel through stages
+        for i in range(1, self.n_stages):
+            if state.interrupted: break
+
+            state.job = f'{i}/{self.n_frames}'
+            state.job_no = i + 1
+
+            # only change target prompts
+            switch_to_stage(i)
+            with on_cfg_denoiser_wrapper(partial(get_cond_callback, [to_pos_hidden])):
+                if self.genesis == Gensis.FIXED:
+                    imgs = process_p(append=False)      # stash it to make order right
+                elif self.genesis == Gensis.SUCCESSIVE:
+                    with p_steps_overrider(self.p, steps=1):    # ignore final image, only need cond
+                        process_p(save=False, append=False)
+                else: raise ValueError(f'invalid genesis: {self.genesis.value}')
+
+            # ========== ↓↓↓ major differences from run_linear() ↓↓↓ ==========
+            
+            # decide change portion in each iter
+            L1 = torch.abs(from_pos_hidden.value - to_pos_hidden.value)
+            if   self.replace_dim == ModeReplaceDim.RANDOM:
+                dist = L1                  # [T=77, D=768]
+            elif self.replace_dim == ModeReplaceDim.TOKEN:
+                dist = L1.mean(axis=1)     # [T=77]
+            elif self.replace_dim == ModeReplaceDim.CHANNEL:
+                dist = L1.mean(axis=0)     # [D=768]
+            else: raise ValueError(f'unknown replace_dim: {self.replace_dim}')
+            mask = dist > EPS
+            dist = torch.where(mask, dist, 0.0)
+            n_diff = mask.sum().item()            # when value differs we have mask==True
+            n_inter = self.steps[i] + 1
+            replace_count = int(n_diff / n_inter) + 1    # => accumulative modifies [1/T, 2/T, .. T-1/T] of total cond
+
+            # Step 2: draw the replaced images
+            inter_pos_hidden.value = from_pos_hidden.value
+            is_break_iter = False
+            for _ in range(1, n_inter):
+                if state.interrupted: is_break_iter = True ; break
+
+                inter_pos_hidden.value = replace_until_match(inter_pos_hidden.value, to_pos_hidden.value, replace_count, dist=dist, order=self.replace_order)
+                with on_cfg_denoiser_wrapper(partial(set_cond_callback, [inter_pos_hidden])):
+                    process_p()
+            
+            # ========== ↑↑↑ major differences from run_linear() ↑↑↑ ==========
+
+            if is_break_iter: break
+
+            # Step 3: append the final stage
+            if self.genesis != Gensis.SUCCESSIVE: self.images.extend(imgs)
+            # move to next stage
+            from_pos_hidden.value = to_pos_hidden.value
+            inter_pos_hidden.value = None
+
+    ''' ↓↓↓ extension support ↓↓↓ '''
+
+    def ext_depth_preprocess(self, p:Processing, depth_img:PILImage):  # copy from repo `AnonymousCervine/depth-image-io-for-SDWebui`
+        from types import MethodType
+        from einops import repeat, rearrange
+        import modules.shared as shared
+        import modules.devices as devices
+
+        def sanitize_pil_image_mode(img):
+            if img.mode in {'P', 'CMYK', 'HSV'}:
+                img = img.convert(mode='RGB')
+            return img
+
+        def alt_depth_image_conditioning(self, source_image):
+            with devices.autocast():
+                conditioning_image = self.sd_model.get_first_stage_encoding(self.sd_model.encode_first_stage(source_image))
+            depth_data = np.array(sanitize_pil_image_mode(depth_img))
+
+            if len(np.shape(depth_data)) == 2:
+                depth_data = rearrange(depth_data, "h w -> 1 1 h w")
+            else:
+                depth_data = rearrange(depth_data, "h w c -> c 1 1 h w")[0]
+            depth_data = torch.from_numpy(depth_data).to(device=shared.device).to(dtype=torch.float32)
+            depth_data = repeat(depth_data, "1 ... -> n ...", n=self.batch_size)
+            
+            conditioning = torch.nn.functional.interpolate(
+                depth_data,
+                size=conditioning_image.shape[2:],
+                mode="bicubic",
+                align_corners=False,
+            )
+            (depth_min, depth_max) = torch.aminmax(conditioning)
+            conditioning = 2. * (conditioning - depth_min) / (depth_max - depth_min) - 1.
+            return conditioning
+        
+        p.depth2img_image_conditioning = MethodType(alt_depth_image_conditioning, p)
+        
+        def alt_txt2img_image_conditioning(self, x, width=None, height=None):
+            fake_img = torch.zeros(1, 3, height or self.height, width or self.width).to(shared.device).type(self.sd_model.dtype)
+            return self.depth2img_image_conditioning(fake_img)
+
+        p.txt2img_image_conditioning = MethodType(alt_txt2img_image_conditioning, p)
+
+    def ext_depth_postprocess(self, p:Processing, depth_img:PILImage):
+        depth_img.close()

tools/README.txt

@@ -0,0 +1,25 @@
+Put your post-processing tools or linkings here.
+
+The directory layout should be like:
+
+tools
+├── install.cmd
+├── link.cmd
+├── busybox.exe
+├── realesrgan-ncnn-vulkan
+│   ├── realesrgan-ncnn-vulkan.exe      # executable
+│   └── models                          # model checkpoints
+│       ├── *.bin
+│       ├── *.param
+│       └── *.pth
+├── rife-ncnn-vulkan
+│   ├── rife-ncnn-vulkan.exe            # executable
+│   └── rife*                           # model checkpoints
+│       ├── *.bin
+│       ├── *.param
+│       └── *.pth
+└── ffmpeg
+    └── bin
+        ├── ffmpeg.exe                  # executable
+        ├── ffplay.exe
+        └── ffprobe.exe

tools/install.cmd

@@ -0,0 +1,109 @@
+@REM Auto download and setup post-process tools
+@ECHO OFF
+SETLOCAL
+
+REM Usage: install.cmd        install and keep .downloaded folder
+REM        install.cmd -c     install and clean .downloaded folder
+
+TITLE Install tools for post-process...
+CD %~dp0
+
+REM paths to web resources
+SET CURL_BIN=curl.exe -L -C -
+
+SET BBOX_URL=https://frippery.org/files/busybox/busybox.exe
+SET BBOX_BIN=busybox.exe
+SET UNZIP_BIN=%BBOX_BIN% unzip
+
+SET RESR_URL=https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-windows.zip
+SET RESR_ZIP=realesrgan-ncnn-vulkan.zip
+SET RESR_DIR=realesrgan-ncnn-vulkan
+
+SET RIFE_URL=https://github.com/nihui/rife-ncnn-vulkan/releases/download/20221029/rife-ncnn-vulkan-20221029-windows.zip
+SET RIFE_ZIP=rife-ncnn-vulkan.zip
+SET RIFE_DIR=rife-ncnn-vulkan
+SET RIFE_RDIR=rife-ncnn-vulkan-20221029-windows
+
+SET FFMPEG_URL=https://github.com/GyanD/codexffmpeg/releases/download/5.1.2/ffmpeg-5.1.2-full_build-shared.zip
+SET FFMPEG_ZIP=ffmpeg.zip
+SET FFMPEG_DIR=ffmpeg
+SET FFMPEG_RDIR=ffmpeg-5.1.2-full_build-shared
+
+REM make cache tmpdir
+SET DOWNLOAD_DIR=.download
+IF NOT EXIST %DOWNLOAD_DIR% MKDIR %DOWNLOAD_DIR%
+ATTRIB +H %DOWNLOAD_DIR%
+
+REM start installation
+ECHO ==================================================
+
+ECHO [0/3] download BusyBox
+IF EXIST %BBOX_BIN% GOTO skip_bbox
+%CURL_BIN% %BBOX_URL% -o %BBOX_BIN%
+:skip_bbox
+
+ECHO ==================================================
+
+ECHO [1/3] install Real-ESRGAN
+IF EXIST %RESR_DIR% GOTO skip_resr
+IF EXIST %DOWNLOAD_DIR%\%RESR_ZIP% GOTO skip_dl_resr
+ECHO ^>^> download from %RESR_URL%
+%CURL_BIN% %RESR_URL% -o %DOWNLOAD_DIR%\%RESR_ZIP%
+IF ERRORLEVEL 1 GOTO die
+:skip_dl_resr
+ECHO ^>^> uzip %RESR_ZIP%
+MKDIR %RESR_DIR%
+%UNZIP_BIN% %DOWNLOAD_DIR%\%RESR_ZIP% -d %RESR_DIR%
+IF ERRORLEVEL 1 GOTO die
+:skip_resr
+
+ECHO ==================================================
+
+ECHO [2/3] install RIFE
+IF EXIST %RIFE_DIR% GOTO skip_rife
+IF EXIST %DOWNLOAD_DIR%\%RIFE_ZIP% GOTO skip_dl_rife
+ECHO ^>^> download from %RIFE_URL%
+%CURL_BIN% %RIFE_URL% -o %DOWNLOAD_DIR%\%RIFE_ZIP%
+IF ERRORLEVEL 1 GOTO die
+:skip_dl_rife
+ECHO ^>^> uzip %RIFE_ZIP%
+%UNZIP_BIN% %DOWNLOAD_DIR%\%RIFE_ZIP%
+IF ERRORLEVEL 1 GOTO die
+RENAME %RIFE_RDIR% %RIFE_DIR%
+:skip_rife
+
+ECHO ==================================================
+
+ECHO [3/3] install FFmpeg
+IF EXIST %FFMPEG_DIR% GOTO skip_ffmpeg
+IF EXIST %DOWNLOAD_DIR%\%FFMPEG_ZIP% GOTO skip_dl_ffmpeg
+ECHO ^>^> download from %FFMPEG_URL%
+%CURL_BIN% %FFMPEG_URL% -o %DOWNLOAD_DIR%\%FFMPEG_ZIP%
+IF ERRORLEVEL 1 GOTO die
+:skip_dl_ffmpeg
+ECHO ^>^> uzip %FFMPEG_ZIP%
+%UNZIP_BIN% %DOWNLOAD_DIR%\%FFMPEG_ZIP%
+IF ERRORLEVEL 1 GOTO die
+RENAME %FFMPEG_RDIR% %FFMPEG_DIR%
+:skip_ffmpeg
+
+ECHO ==================================================
+
+REM clean cache
+IF /I "%~1"=="-c" (
+  ATTRIB -H %DOWNLOAD_DIR%
+  RMDIR /S /Q %DOWNLOAD_DIR%
+)
+
+REM finished
+ECHO ^>^> Done!
+ECHO.
+GOTO :end
+
+REM error handle
+:die
+ECHO ^<^< Error!
+ECHO ^<^< errorlevel: %ERRORLEVEL%
+
+:end
+PAUSE

tools/link.cmd

@@ -0,0 +1,20 @@
+@REM Make soft links to post-process tools
+@ECHO OFF
+SETLOCAL
+
+SET RESR_HOME=D:\tools\realesrgan-ncnn-vulkan
+SET RIFE_HOME=D:\tools\rife-ncnn-vulkan
+SET FFMPEG_HOME=D:\tools\ffmpeg
+
+@ECHO ON
+
+PUSHD %~dp0
+MKLINK /J realesrgan-ncnn-vulkan %RESR_HOME%
+MKLINK /J rife-ncnn-vulkan       %RIFE_HOME%
+MKLINK /J ffmpeg                 %FFMPEG_HOME%
+POPD
+
+ECHO ^>^> Done!
+ECHO.
+
+PAUSE