diff --git a/.gitattributes b/.gitattributes
index a6344aac8c09253b3b630fb776ae94478aa0275b..15a45ca35101f815b14831e73f8ef255895a9ea0 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.gif filter=lfs diff=lfs merge=lfs -text
+*.mp4 filter=lfs diff=lfs merge=lfs -text
+*.wav filter=lfs diff=lfs merge=lfs -text
+*.jpg filter=lfs diff=lfs merge=lfs -text
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000000000000000000000000000000000000..c9d60913b5aee2408c17a596bc65d10ad84444a1
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,14 @@
+__pycache__
+.idea
+*.pyc
+.DS_Store
+checkpoints
+results
+venv
+*.egg-info
+build
+dist
+*.eg
+checkpoints_test
+logs
+third_party
\ No newline at end of file
diff --git a/DockerfileAPI b/DockerfileAPI
new file mode 100644
index 0000000000000000000000000000000000000000..ccf2440704f98fc27f1d83de1dfb73bb0848b681
--- /dev/null
+++ b/DockerfileAPI
@@ -0,0 +1,7 @@
+FROM shaoguo/faster_liveportrait:v3
+USER root
+RUN mkdir -p /root/FasterLiveportrait
+RUN chown -R /root/FasterLiveportrait
+COPY . /root/FasterLiveportrait
+WORKDIR /root/FasterLiveportrait
+CMD ["/bin/bash && bash scripts/start_api.sh"]
\ No newline at end of file
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 0000000000000000000000000000000000000000..510d38feced882e28adafaf80d4fc473a87a9ccc
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,33 @@
+MIT License
+
+Copyright (c) 2025 warmshao
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+---
+
+ADDITIONAL NOTICE FOR MODELS:
+
+This repository may contain or reference machine learning models. These models
+are subject to their respective licenses, which may differ from the MIT license
+applied to the code in this repository. Users are responsible for complying
+with the license terms of any models they use. This repository and its
+maintainers assume no responsibility for model licensing compliance.
+
+Please check the original source and license of each model before use.
diff --git a/README.md b/README.md
index 4193dac335c6da4994b4d8ef4945537d2d387ea3..dfc8901e548b485c0b7cf9e8a6192ff2b4b497ec 100644
--- a/README.md
+++ b/README.md
@@ -1,11 +1,183 @@
----
-title: FasterLivepotrait
-emoji: 💻
-colorFrom: purple
-colorTo: gray
-sdk: docker
-pinned: false
-license: mit
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+## FasterLivePortrait: Bring portraits to life in Real Time!
+English | 中文
+
+**Original repository: [LivePortrait](https://github.com/KwaiVGI/LivePortrait), thanks to the authors for sharing**
+
+**New features:**
+* Achieved real-time running of LivePortrait on RTX 3090 GPU using TensorRT, reaching speeds of 30+ FPS. This is the speed for rendering a single frame, including pre- and post-processing, not just the model inference speed.
+* Seamless support for native gradio app, with several times faster speed and support for simultaneous inference on multiple faces and Animal Model.
+* Added support for [JoyVASA](https://github.com/jdh-algo/JoyVASA), which can drive videos or images with audio.
+
+**If you find this project useful, please give it a star ✨✨**
+
+### Demo (Explore more features)
+* Anyone want this? Fell free to contact me.
+
+
+
+
+* Text-driven video, based on kokoro-82M:
+
+
+
+* Audio-driven video (real-time):
+
+
+
+* Animal-driven:
+
+
+
+* Multiple faces driven simultaneously:
+
+
+
+
+### Environment Setup
+* Option 1 (recommended): If you are a Windows user, you can directly download the [integrated package](https://github.com/warmshao/FasterLivePortrait/releases/tag/v1.8).
+ * You need to install [git](https://git-scm.com/downloads) first, then double-click `update.bat` to update the code.
+ * Double-click `scripts/all_onnx2trt.bat` to convert onnx files to tensorrt files.
+ * Double-click `webui.bat` to open the webpage, or double-click `camera.bat` to open the camera for real-time operation.
+* Option 2: Docker.A docker image is provided for eliminating the need to install onnxruntime-gpu and TensorRT manually.
+ * Install [Docker](https://docs.docker.com/desktop/install/windows-install/) according to your system
+ * Download the image: `docker pull shaoguo/faster_liveportrait:v3`
+ * Execute the command, replace `$FasterLivePortrait_ROOT` with the local directory where you downloaded FasterLivePortrait:
+ ```shell
+ docker run -it --gpus=all \
+ --name faster_liveportrait \
+ -v $FasterLivePortrait_ROOT:/root/FasterLivePortrait \
+ --restart=always \
+ -p 9870:9870 \
+ shaoguo/faster_liveportrait:v3 \
+ /bin/bash
+ ```
+* Option 3: Create a new Python virtual environment and install the necessary Python packages manually.
+ * First, install [ffmpeg](https://www.ffmpeg.org/download.html)
+ * Run `pip install -r requirements.txt`
+ * Then follow the tutorials below to install onnxruntime-gpu or TensorRT. Note that this has only been tested on Linux systems.
+
+### Usage
+#### 1. TensorRT Inference(Recommended)
+* (Ignored in Docker) Install TensorRT 8.x (versions >=10.x are not compatible). Remember the installation path of [TensorRT](https://developer.nvidia.com/tensorrt).
+* (Ignored in Docker) Install the grid_sample TensorRT plugin, as the model uses grid sample that requires 5D input, which is not supported by the native grid_sample operator.
+ * `git clone https://github.com/SeanWangJS/grid-sample3d-trt-plugin`
+ * Modify line 30 in `CMakeLists.txt` to: `set_target_properties(${PROJECT_NAME} PROPERTIES CUDA_ARCHITECTURES "60;70;75;80;86")`
+ * `export PATH=/usr/local/cuda/bin:$PATH`
+ * `mkdir build && cd build`
+ * `cmake .. -DTensorRT_ROOT=$TENSORRT_HOME`, replace $TENSORRT_HOME with your own TensorRT root directory.
+ * `make`, remember the address of the .so file, replace `/opt/grid-sample3d-trt-plugin/build/libgrid_sample_3d_plugin.so` in `scripts/onnx2trt.py` and `src/models/predictor.py` with your own .so file path
+* Download ONNX model files:`huggingface-cli download warmshao/FasterLivePortrait --local-dir ./checkpoints`. Convert all ONNX models to TensorRT, run `sh scripts/all_onnx2trt.sh` and `sh scripts/all_onnx2trt_animal.sh`
+* Test the pipeline using tensorrt:
+ ```shell
+ python run.py \
+ --src_image assets/examples/source/s10.jpg \
+ --dri_video assets/examples/driving/d14.mp4 \
+ --cfg configs/trt_infer.yaml
+* To run in real-time using a camera:
+ ```shell
+ python run.py \
+ --src_image assets/examples/source/s10.jpg \
+ --dri_video 0 \
+ --cfg configs/trt_infer.yaml \
+ --realtime
+ ```
+
+#### 2. Onnxruntime Inference
+* First, download the converted onnx model files:`huggingface-cli download warmshao/FasterLivePortrait --local-dir ./checkpoints`.
+* (Ignored in Docker)If you want to use onnxruntime cpu inference, simply `pip install onnxruntime`. However, cpu inference is extremely slow and not recommended. The latest onnxruntime-gpu still doesn't support grid_sample cuda, but I found a branch that supports it. Follow these steps to install `onnxruntime-gpu` from source:
+ * `git clone https://github.com/microsoft/onnxruntime`
+ * `git checkout liqun/ImageDecoder-cuda`. Thanks to liqun for the grid_sample with cuda implementation!
+ * Run the following commands to compile, changing `cuda_version` and `CMAKE_CUDA_ARCHITECTURES` according to your machine (your cuDNN version must be 8.x, 9.x is not compatible):
+ ```shell
+ ./build.sh --parallel \
+ --build_shared_lib --use_cuda \
+ --cuda_version 11.8 \
+ --cuda_home /usr/local/cuda --cudnn_home /usr/local/cuda/ \
+ --config Release --build_wheel --skip_tests \
+ --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES="60;70;75;80;86" \
+ --cmake_extra_defines CMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
+ --disable_contrib_ops \
+ --allow_running_as_root
+ ```
+ * `pip install build/Linux/Release/dist/onnxruntime_gpu-1.17.0-cp310-cp310-linux_x86_64.whl`
+* Test the pipeline using onnxruntime:
+ ```
+ python run.py \
+ --src_image assets/examples/source/s10.jpg \
+ --dri_video assets/examples/driving/d14.mp4 \
+ --cfg configs/onnx_infer.yaml
+ ```
+
+
+### Gradio WebUI
+* onnxruntime: `python webui.py --mode onnx`
+* tensorrt: `python webui.py --mode trt`
+* The default port is 9870. Open the webpage: `http://localhost:9870/`
+
+Hotkeys for webcam mode (when render window is on focus)\
+Q > exit\
+S > Stitching\
+Z > RelativeMotion\
+X > AnimationRegion\
+C > CropDrivingVideo\
+K,L > AdjustSourceScale\
+N,M > AdjustDriverScale
+
+## License
+
+- **Code**: This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+- **Models**: Any machine learning models used in this project are subject to their respective licenses. Please refer to the original model sources for license information. We do not take responsibility for model license compliance.
+
+
+**Changelog**
+- [x] **2025/06/29:** LivePortrait animal v1.1 onnx models are available. Download from [this](https://huggingface.co/warmshao/FasterLivePortrait/tree/main/liveportrait_animal_onnx_v1.1).
+- [x] **2024/12/22:** Add API Deployment `python api.py`, For more information, please refer to the [tutorial](assets/docs/API.md).
+- [x] **2024/12/21:** Added support for [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M), enabling text-driven video or image generation.
+ - Updated code: `git pull origin master` and install the latest Python dependencies `pip install requirements.txt`, or simply double-click `update.bat` on Windows.
+ - Download the model: `huggingface-cli download hexgrad/Kokoro-82M --local-dir .\checkpoints\Kokoro-82M`.
+ - For Linux, install `espeak-ng`: `apt-get -qq -y install espeak-ng > /dev/null 2>&1`
+ - For Windows, refer to [manual installation instructions](https://huggingface.co/hexgrad/Kokoro-82M/discussions/12) and configure the `espeak-ng` environment variables. The current read location is [here](src/pipelines/gradio_live_portrait_pipeline.py:437); modify it if your installation path differs.
+ - Now you can use it normally in the "Drive Text" tab.
+- [x] **2024/12/16:** Added support for [JoyVASA](https://github.com/jdh-algo/JoyVASA), which can drive videos or images with audio. Very cool!
+ - Update code, then download the models: `huggingface-cli download TencentGameMate/chinese-hubert-base --local-dir .\checkpoints\chinese-hubert-base` and `huggingface-cli download jdh-algo/JoyVASA --local-dir ./checkpoints/JoyVASA`
+ - After launching the webui, follow the tutorial below. When the source is a video, it's recommended to only drive the mouth movements
+
+
+
+- [x] **2024/12/14:** Added pickle and image driving, as well as region driving animation_region.
+ - Please update the latest code. Windows users can directly double-click `update.bat` to update, but note that your local code will be overwritten.
+ - Running `python run.py` now automatically saves the corresponding pickle to the same directory as the driving video, allowing for direct reuse.
+ - After opening webui, you can experience the new pickle and image driving, as well as the region driving animation_region features. Note that for image driving, remember to disable `relative motion`.
+- [x] **2024/08/11:** Optimized paste_back speed and fixed some bugs.
+ - Used torchgeometry + cuda to optimize the paste_back function, significantly improving speed. Example: `python run.py --src_image assets/examples/source/s39.jpg --dri_video assets/examples/driving/d0.mp4 --cfg configs/trt_infer.yaml --paste_back --animal`
+ - Fixed issues with Xpose ops causing errors on some GPUs and other bugs. Please use the latest docker image: `docker pull shaoguo/faster_liveportrait:v3`
+- [x] **2024/08/11:** Optimized paste_back speed and fixed some bugs.
+ - Used torchgeometry + cuda to optimize the paste_back function, significantly improving speed. Example: `python run.py --src_image assets/examples/source/s39.jpg --dri_video assets/examples/driving/d0.mp4 --cfg configs/trt_infer.yaml --paste_back --animal`
+ - Fixed issues with Xpose ops causing errors on some GPUs and other bugs. Please use the latest docker image: `docker pull shaoguo/faster_liveportrait:v3`
+- [x] **2024/08/07:** Added support for animal models and MediaPipe models, so you no longer need to worry about copyright issues.
+ - Added support for animal models.
+ - Download the animal ONNX file: `huggingface-cli download warmshao/FasterLivePortrait --local-dir ./checkpoints`, then convert it to TRT format.
+ - Update the Docker image: `docker pull shaoguo/faster_liveportrait:v3`. Using animal model:`python run.py --src_image assets/examples/source/s39.jpg --dri_video 0 --cfg configs/trt_infer.yaml --realtime --animal`
+ - Windows users can download the latest [Windows all-in-one package](https://github.com/warmshao/FasterLivePortrait/releases) from the release page, then unzip and use it.
+ - Simple usage tutorial:
+
+
+
+ - Using MediaPipe model to replace InsightFace
+ - For web usage: `python webui.py --mode trt --mp` or `python webui.py --mode onnx --mp`
+ - For local webcam: `python run.py --src_image assets/examples/source/s12.jpg --dri_video 0 --cfg configs/trt_mp_infer.yaml`
+- [x] **2024/07/24:** Windows integration package, no installation required, one-click run, supports TensorRT and OnnxruntimeGPU. Thanks to @zhanghongyong123456 for their contribution in this [issue](https://github.com/warmshao/FasterLivePortrait/issues/22).
+ - [Optional] If you have already installed CUDA and cuDNN on your Windows computer, please skip this step. I have only verified on CUDA 12.2. If you haven't installed CUDA or encounter CUDA-related errors, you need to follow these steps:
+ - Download [CUDA 12.2](https://developer.nvidia.com/cuda-12-2-0-download-archive?target_os=Windows&target_arch=x86_64), double-click the exe and install following the default settings step by step.
+ - Download the [cuDNN](https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-windows-x86_64-8.9.7.29_cuda12-archive.zip) zip file, extract it, and copy the lib, bin, and include folders from the cuDNN folder to the CUDA 12.2 folder (default is C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2)
+ - Download the installation-free [Windows integration package](https://github.com/warmshao/FasterLivePortrait/releases) from the release page and extract it.
+ - Enter `FasterLivePortrait-windows` and double-click `scripts/all_onnx2trt.bat` to convert onnx files, which will take some time.
+ - For web demo: Double-click `webui.bat`, open the webpage: `http://localhost:9870/`
+ - For real-time camera operation, double-click `camera.bat`，press `q` to stop. If you want to change the target image, run in command line: `camera.bat assets/examples/source/s9.jpg`
+- [x] **2024/07/18:** macOS support added(No need for Docker, Python is enough). M1/M2 chips are faster, but it's still quite slow 😟
+ - Install ffmpeg: `brew install ffmpeg`
+ - Set up a Python 3.10 virtual environment. Recommend using [miniforge](https://github.com/conda-forge/miniforge): `conda create -n flip python=3.10 && conda activate flip`
+ - Install requirements: `pip install -r requirements_macos.txt`
+ - Download ONNX files: `huggingface-cli download warmshao/FasterLivePortrait --local-dir ./checkpoints`
+ - Test: `python webui.py --mode onnx`
+- [x] **2024/07/17:** Added support for Docker environment, providing a runnable image.
diff --git a/README_ZH.md b/README_ZH.md
new file mode 100644
index 0000000000000000000000000000000000000000..2000b3896de7e2b6e02a33b4ccf23163c5bda5e0
--- /dev/null
+++ b/README_ZH.md
@@ -0,0 +1,173 @@
+## FasterLivePortrait：Bring portrait to life in Real Time!
+English | 中文
+
+**原仓库: [LivePortrait](https://github.com/KwaiVGI/LivePortrait)，感谢作者的分享**
+
+**新增功能：**
+* 通过TensorRT实现在RTX 3090显卡上**实时**运行LivePortrait，速度达到 30+ FPS. 这个速度是实测渲染出一帧的速度，而不仅仅是模型的推理时间。
+* 无缝支持原生的gradio app, 速度快了好几倍，同时支持多张人脸、Animal模型。
+* 增加[JoyVASA](https://github.com/jdh-algo/JoyVASA)的支持，可以用音频驱动视频或图片。
+
+**如果你觉得这个项目有用，帮我点个star吧✨✨**
+
+### Demo(还有很多功能等你探索)
+* 文本驱动视频，基于kokoro-82M:
+
+
+* 声音驱动视频(可以实时):
+
+
+* 动物驱动:
+
+
+* 多张人脸同时驱动:
+
+
+
+
+### 环境安装
+* 方式1：如果你是Windows用户，推荐可以直接下载[整合包](https://github.com/warmshao/FasterLivePortrait/releases/tag/v1.8)。
+ * 需要先安装好[git](https://git-scm.com/downloads), 双击`update.bat`更新代码。
+ * 双击`scripts/all_onnx2trt.bat`转换onnx文件为tensorrt文件。
+ * 双击`webui.bat`打开网页，或者双击`camera.bat`打开摄像头实时运行。
+* 方式2：Docker，提供了一个镜像，不用再自己安装onnxruntime-gpu和TensorRT。
+ * 根据自己的系统安装[docker](https://docs.docker.com/desktop/install/windows-install/)
+ * 下载镜像：`docker pull shaoguo/faster_liveportrait:v3`
+ * 执行命令, `$FasterLivePortrait_ROOT`要替换成你下载的FasterLivePortrait在本地的目录:
+ ```shell
+ docker run -it --gpus=all \
+ --name faster_liveportrait \
+ -v $FasterLivePortrait_ROOT:/root/FasterLivePortrait \
+ --restart=always \
+ -p 9870:9870 \
+ shaoguo/faster_liveportrait:v3 \
+ /bin/bash
+ ```
+ * 然后可以根据下面Onnxruntime 推理和TensorRT 推理教程进行使用。
+
+* 方式3：新建一个python虚拟环境，自己安装必要的python包
+ * 请先安装[ffmpeg](https://www.ffmpeg.org/download.html)
+ * `pip install -r requirements.txt`
+ * 再根据以下教程安装onnxruntime-gpu或TensorRT。
+
+### 使用方法
+#### 1. TensorRT 推理(推荐, 可以实时)
+* (Docker环境可忽略）安装TensorRT，请记住[TensorRT](https://developer.nvidia.com/tensorrt)安装的路径。
+* (Docker环境可忽略）安装 grid_sample的tensorrt插件，因为模型用到的grid sample需要有5d的输入,原生的grid_sample 算子不支持。
+ * `git clone https://github.com/SeanWangJS/grid-sample3d-trt-plugin`
+ * 修改`CMakeLists.txt`中第30行为:`set_target_properties(${PROJECT_NAME} PROPERTIES CUDA_ARCHITECTURES "60;70;75;80;86")`
+ * `export PATH=/usr/local/cuda/bin:$PATH`
+ * `mkdir build && cd build`
+ * `cmake .. -DTensorRT_ROOT=$TENSORRT_HOME`,$TENSORRT_HOME 替换成你自己TensorRT的根目录。
+ * `make`，记住so文件的地址，将`scripts/onnx2trt.py`和`src/models/predictor.py`里`/opt/grid-sample3d-trt-plugin/build/libgrid_sample_3d_plugin.so`替换成自己的so路径
+* 下载Onnx文件：`huggingface-cli download warmshao/FasterLivePortrait --local-dir ./checkpoints`。将onnx模型转为tensorrt，运行`sh scripts/all_onnx2trt.sh`和`sh scripts/all_onnx2trt_animal.sh`
+* 用tensorrt测试pipeline：
+ ```shell
+ python run.py \
+ --src_image assets/examples/source/s10.jpg \
+ --dri_video assets/examples/driving/d14.mp4 \
+ --cfg configs/trt_infer.yaml
+ ```
+ 如果要使用摄像头实时运行：
+ ```shell
+ python run.py \
+ --src_image assets/examples/source/s10.jpg \
+ --dri_video 0 \
+ --cfg configs/trt_infer.yaml \
+ --realtime
+ ```
+#### 2. Onnxruntime 推理
+* 首先下载我转换好的[模型onnx文件](https://huggingface.co/warmshao/FasterLivePortrait): `huggingface-cli download warmshao/FasterLivePortrait --local-dir ./checkpoints`。
+* (Docker环境可忽略）如果你要用onnxruntime cpu推理的话，直接`pip install onnxruntime`即可，但是cpu推理超级慢。但是最新的onnxruntime-gpu仍然无法支持grid_sample cuda，好在我看到一位大佬在分支上支持了，按照以下步骤源码安装`onnxruntime-gpu`:
+ * `git clone https://github.com/microsoft/onnxruntime`
+ * `git checkout liqun/ImageDecoder-cuda`. Thanks for liqun's grid_sample with cuda implementation!
+ * 运行以下命令编译,`cuda_version`和`CMAKE_CUDA_ARCHITECTURES`根据自己的机器更改:
+ ```shell
+ ./build.sh --parallel \
+ --build_shared_lib --use_cuda \
+ --cuda_version 11.8 \
+ --cuda_home /usr/local/cuda --cudnn_home /usr/local/cuda/ \
+ --config Release --build_wheel --skip_tests \
+ --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES="60;70;75;80;86" \
+ --cmake_extra_defines CMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
+ --disable_contrib_ops \
+ --allow_running_as_root
+ ```
+ * `pip install build/Linux/Release/dist/onnxruntime_gpu-1.17.0-cp310-cp310-linux_x86_64.whl`就可以了
+* 用onnxruntime测试pipeline：
+ ```shell
+ python run.py \
+ --src_image assets/examples/source/s10.jpg \
+ --dri_video assets/examples/driving/d14.mp4 \
+ --cfg configs/onnx_infer.yaml
+ ```
+
+### Gradio WebUI
+* onnxruntime: `python webui.py --mode onnx`
+* tensorrt: `python webui.py --mode trt`
+* 默认端口在9870，打开网页：`http://localhost:9870/`
+
+Hotkeys for webcam mode (when render window is on focus)\
+Q > exit\
+S > Stitching\
+Z > RelativeMotion\
+X > AnimationRegion\
+C > CropDrivingVideo\
+K,L > AdjustSourceScale\
+N,M > AdjustDriverScale
+
+## 许可证
+
+- **代码**: 本项目采用 MIT 许可证 - 详细信息请查看 [LICENSE](LICENSE) 文件。
+- **模型**: 本项目中使用的任何机器学习模型均遵循其各自的许可证。请参考原始模型来源获取许可证信息。我们不承担模型许可证合规性的责任。
+
+
+**日志**
+- [x] **2025/06/29:** [LivePortrait animal v1.1 onnx模型](https://huggingface.co/warmshao/FasterLivePortrait/tree/main/liveportrait_animal_onnx_v1.1)。
+- [x] **2024/12/22:** 增加api部署`python api.py`, 其他参考[教程](assets/docs/API_ZH.md)使用。
+- [x] **2024/12/21:** 增加[Kokoro-82M](hhttps://huggingface.co/hexgrad/Kokoro-82M)的支持，可以用文本驱动视频或图片。
+ - 更新代码, `git pull origin master`并安装最新的python依赖 `pip install requirements.txt`， 或者 windows下直接双击 `update.bat`.
+ - 然后下载模型: `huggingface-cli download hexgrad/Kokoro-82M --local-dir .\checkpoints\Kokoro-82M`.
+ - 如果是Linux请安装`apt-get -qq -y install espeak-ng > /dev/null 2>&1`
+ - 如果是windows请参考[自行安装](https://huggingface.co/hexgrad/Kokoro-82M/discussions/12)并配置好`espeak-ng`环境变量。我是在[这里](src/pipelines/gradio_live_portrait_pipeline.py:437)读取，如果你的位置变了，请自行修改。
+ - 然后就可以在Drive Text的标签页正常使用了。
+- [x] **2024/12/16:** 增加[JoyVASA](https://github.com/jdh-algo/JoyVASA)的支持，可以用音频驱动视频或图片。非常酷！
+ - 更新代码，然后下载模型: `huggingface-cli download TencentGameMate/chinese-hubert-base --local-dir .\checkpoints\chinese-hubert-base` 和 ` huggingface-cli download jdh-algo/JoyVASA --local-dir ./checkpoints/JoyVASA`
+ - 启动webui后根据以下教程使用即可，建议source 是视频的情况下只驱动嘴部
+
+
+
+- [x] **2024/12/14:** 增加pickle和image驱动以及区域驱动`animation_region`。
+ - 请更新最新的代码，windows用户可以直接双击`update.bat`更新，但请注意本地的代码将会被覆盖。
+ - `python run.py ` 现在运行 `driving video`会自动保存对应的pickle到跟`driving video`一样的目录，可以直接复用。
+ - 打开`webui`后即可体验新的pickle和image驱动以及区域驱动`animation_region`等功能。注意image驱动记得把`relative motion`取消掉。
+- [x] **2024/08/11:** 优化paste_back的速度，修复一些bug。
+ - 用torchgeometry + cuda优化paste_back函数，现在速度提升了很多。示例：`python run.py --src_image assets/examples/source/s39.jpg --dri_video assets/examples/driving/d0.mp4 --cfg configs/trt_infer.yaml --paste_back --animal`
+ - 修复Xpose的ops在一些显卡运行报错的问题等bug。请使用最新的镜像:`docker pull shaoguo/faster_liveportrait:v3`
+- [x] **2024/08/07:** 增加animal模型的支持，同时支持mediapipe模型，现在你不用再担心版权的问题。
+ - 增加对animal模型的支持。
+ - 需要下载animal的onnx文件：`huggingface-cli download warmshao/FasterLivePortrait --local-dir ./checkpoints`，然后转换成trt文件。
+ - 更新镜像`docker pull shaoguo/faster_liveportrait:v3`, 使用animal模型的示例:`python run.py --src_image assets/examples/source/s39.jpg --dri_video 0 --cfg configs/trt_infer.yaml --realtime --animal`
+ - windows系统可以从release页下载最新的[windows 整合包](https://github.com/warmshao/FasterLivePortrait/releases)，解压后使用。
+ - 简单的使用教程：
+
+
+
+ - 使用mediapipe模型替代insight_face
+ - 网页端使用: `python webui.py --mode trt --mp` 或 `python webui.py --mode onnx --mp`
+ - 本地摄像头运行: `python run.py --src_image assets/examples/source/s12.jpg --dri_video assets/examples/driving/d0.mp4 --cfg configs/trt_mp_infer.yaml`
+- [x] **2024/07/24:** Windows的整合包, 免安装一键运行，支持TensorRT和OnnxruntimeGPU。感谢@zhanghongyong123456在[issue](https://github.com/warmshao/FasterLivePortrait/issues/22)的贡献。
+ - 【可选】如果你的windows电脑已经装过cuda和cudnn，请忽略这一步。我只在cuda12.2上验证过，如果没安装cuda或报cuda相关的错，你需要按照以下步骤进行安装：
+ - 下载[cuda12.2](https://developer.nvidia.com/cuda-12-2-0-download-archive?target_os=Windows&target_arch=x86_64), 双击exe后按照默认设置一步步安装即可。
+ - 下载[cudnn](https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-windows-x86_64-8.9.7.29_cuda12-archive.zip) 压缩包，解压后将cudnn 文件夹下的lib、bin、include 文件夹复制到 CUDA12.2 文件夹下（默认为C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2）
+ - 从release页下载免安装[windows 整合包](https://github.com/warmshao/FasterLivePortrait/releases)并解压。
+ - 进入`FasterLivePortrait-windows`后双击`scripts/all_onnx2trt.bat`对onnx文件进行转换，这会等上一段时间。
+ - 网页端demo：双击`webui.bat`, 打开网页：`http://localhost:9870/`
+ - 摄像头实时运行，双击`camera.bat`，按`q`停止。如果你想更换目标图像，命令行运行:`camera.bat assets/examples/source/s9.jpg`。
+- [x] **2024/07/18:** MacOS支持(不需要Docker，python就可以了），M1/M2的速度比较快，但还是很慢😟
+ - 安装ffmpeg: `brew install ffmpeg`
+ - 安装python=3.10的虚拟环境，推荐可以用[miniforge](https://github.com/conda-forge/miniforge).`conda create -n flip python=3.10 && conda activate flip`
+ - `pip install -r requirements_macos.txt`
+ - 下载onnx文件: `huggingface-cli download warmshao/FasterLivePortrait --local-dir ./checkpoints`
+ - 测试: `python webui.py --mode onnx`
+- [x] **2024/07/17:** 增加docker环境的支持，提供可运行的镜像。
diff --git a/api.py b/api.py
new file mode 100644
index 0000000000000000000000000000000000000000..4a1c7157edeeb8c1c3ec243f778f16fcda376a61
--- /dev/null
+++ b/api.py
@@ -0,0 +1,479 @@
+# -*- coding: utf-8 -*-
+# @Time : 2024/9/13 0:23
+# @Project : FasterLivePortrait
+# @FileName: api.py
+import pdb
+import shutil
+from typing import Optional, Dict, Any
+import io
+import os
+import subprocess
+import uvicorn
+import cv2
+import time
+import numpy as np
+import os
+import datetime
+import platform
+import pickle
+from tqdm import tqdm
+from pydantic import BaseModel
+from fastapi import APIRouter, Depends, FastAPI, Request, Response, UploadFile
+from fastapi import File, Body, Form
+from omegaconf import OmegaConf
+from fastapi.responses import StreamingResponse
+from zipfile import ZipFile
+from src.pipelines.faster_live_portrait_pipeline import FasterLivePortraitPipeline
+from src.utils.utils import video_has_audio
+from src.utils import logger
+
+# model dir
+project_dir = os.path.dirname(__file__)
+checkpoints_dir = os.environ.get("FLIP_CHECKPOINT_DIR", os.path.join(project_dir, "checkpoints"))
+log_dir = os.path.join(project_dir, "logs")
+os.makedirs(log_dir, exist_ok=True)
+result_dir = os.path.join(project_dir, "results")
+os.makedirs(result_dir, exist_ok=True)
+
+logger_f = logger.get_logger("faster_liveportrait_api", log_file=os.path.join(log_dir, "log_run.log"))
+
+app = FastAPI()
+
+global pipe
+
+if platform.system().lower() == 'windows':
+ FFMPEG = "third_party/ffmpeg-7.0.1-full_build/bin/ffmpeg.exe"
+else:
+ FFMPEG = "ffmpeg"
+
+
+def check_all_checkpoints_exist(infer_cfg):
+ """
+ check whether all checkpoints exist
+ :return:
+ """
+ ret = True
+ for name in infer_cfg.models:
+ if not isinstance(infer_cfg.models[name].model_path, str):
+ for i in range(len(infer_cfg.models[name].model_path)):
+ infer_cfg.models[name].model_path[i] = infer_cfg.models[name].model_path[i].replace("./checkpoints",
+ checkpoints_dir)
+ if not os.path.exists(infer_cfg.models[name].model_path[i]) and not os.path.exists(
+ infer_cfg.models[name].model_path[i][:-4] + ".onnx"):
+ return False
+ else:
+ infer_cfg.models[name].model_path = infer_cfg.models[name].model_path.replace("./checkpoints",
+ checkpoints_dir)
+ if not os.path.exists(infer_cfg.models[name].model_path) and not os.path.exists(
+ infer_cfg.models[name].model_path[:-4] + ".onnx"):
+ return False
+ for name in infer_cfg.animal_models:
+ if not isinstance(infer_cfg.animal_models[name].model_path, str):
+ for i in range(len(infer_cfg.animal_models[name].model_path)):
+ infer_cfg.animal_models[name].model_path[i] = infer_cfg.animal_models[name].model_path[i].replace(
+ "./checkpoints",
+ checkpoints_dir)
+ if not os.path.exists(infer_cfg.animal_models[name].model_path[i]) and not os.path.exists(
+ infer_cfg.animal_models[name].model_path[i][:-4] + ".onnx"):
+ return False
+ else:
+ infer_cfg.animal_models[name].model_path = infer_cfg.animal_models[name].model_path.replace("./checkpoints",
+ checkpoints_dir)
+ if not os.path.exists(infer_cfg.animal_models[name].model_path) and not os.path.exists(
+ infer_cfg.animal_models[name].model_path[:-4] + ".onnx"):
+ return False
+
+ # XPOSE
+ xpose_model_path = os.path.join(checkpoints_dir, "liveportrait_animal_onnx/xpose.pth")
+ if not os.path.exists(xpose_model_path):
+ return False
+ embeddings_cache_9_path = os.path.join(checkpoints_dir, "liveportrait_animal_onnx/clip_embedding_9.pkl")
+ if not os.path.exists(embeddings_cache_9_path):
+ return False
+ embeddings_cache_68_path = os.path.join(checkpoints_dir, "liveportrait_animal_onnx/clip_embedding_68.pkl")
+ if not os.path.exists(embeddings_cache_68_path):
+ return False
+ return ret
+
+
+def convert_onnx_to_trt_models(infer_cfg):
+ ret = True
+ for name in infer_cfg.models:
+ if not isinstance(infer_cfg.models[name].model_path, str):
+ for i in range(len(infer_cfg.models[name].model_path)):
+ trt_path = infer_cfg.models[name].model_path[i]
+ onnx_path = trt_path[:-4] + ".onnx"
+ if not os.path.exists(trt_path):
+ convert_cmd = f"python scripts/onnx2trt.py -o {onnx_path}"
+ logger_f.info(f"convert onnx model: {onnx_path}")
+ result = subprocess.run(convert_cmd, shell=True, check=True)
+ # 检查结果
+ if result.returncode == 0:
+ logger_f.info(f"convert onnx model: {onnx_path} successful")
+ else:
+ logger_f.error(f"convert onnx model: {onnx_path} failed")
+ return False
+ else:
+ trt_path = infer_cfg.models[name].model_path
+ onnx_path = trt_path[:-4] + ".onnx"
+ if not os.path.exists(trt_path):
+ convert_cmd = f"python scripts/onnx2trt.py -o {onnx_path}"
+ logger_f.info(f"convert onnx model: {onnx_path}")
+ result = subprocess.run(convert_cmd, shell=True, check=True)
+ # 检查结果
+ if result.returncode == 0:
+ logger_f.info(f"convert onnx model: {onnx_path} successful")
+ else:
+ logger_f.error(f"convert onnx model: {onnx_path} failed")
+ return False
+
+ for name in infer_cfg.animal_models:
+ if not isinstance(infer_cfg.animal_models[name].model_path, str):
+ for i in range(len(infer_cfg.animal_models[name].model_path)):
+ trt_path = infer_cfg.animal_models[name].model_path[i]
+ onnx_path = trt_path[:-4] + ".onnx"
+ if not os.path.exists(trt_path):
+ convert_cmd = f"python scripts/onnx2trt.py -o {onnx_path}"
+ logger_f.info(f"convert onnx model: {onnx_path}")
+ result = subprocess.run(convert_cmd, shell=True, check=True)
+ # 检查结果
+ if result.returncode == 0:
+ logger_f.info(f"convert onnx model: {onnx_path} successful")
+ else:
+ logger_f.error(f"convert onnx model: {onnx_path} failed")
+ return False
+ else:
+ trt_path = infer_cfg.animal_models[name].model_path
+ onnx_path = trt_path[:-4] + ".onnx"
+ if not os.path.exists(trt_path):
+ convert_cmd = f"python scripts/onnx2trt.py -o {onnx_path}"
+ logger_f.info(f"convert onnx model: {onnx_path}")
+ result = subprocess.run(convert_cmd, shell=True, check=True)
+ # 检查结果
+ if result.returncode == 0:
+ logger_f.info(f"convert onnx model: {onnx_path} successful")
+ else:
+ logger_f.error(f"convert onnx model: {onnx_path} failed")
+ return False
+ return ret
+
+
+@app.on_event("startup")
+async def startup_event():
+ global pipe
+ # default use trt model
+ cfg_file = os.path.join(project_dir, "configs/trt_infer.yaml")
+ infer_cfg = OmegaConf.load(cfg_file)
+ checkpoints_exist = check_all_checkpoints_exist(infer_cfg)
+
+ # first: download model if not exist
+ if not checkpoints_exist:
+ download_cmd = f"huggingface-cli download warmshao/FasterLivePortrait --local-dir {checkpoints_dir}"
+ logger_f.info(f"download model: {download_cmd}")
+ result = subprocess.run(download_cmd, shell=True, check=True)
+ # 检查结果
+ if result.returncode == 0:
+ logger_f.info(f"Download checkpoints to {checkpoints_dir} successful")
+ else:
+ logger_f.error(f"Download checkpoints to {checkpoints_dir} failed")
+ exit(1)
+ # second: convert onnx model to trt
+ convert_ret = convert_onnx_to_trt_models(infer_cfg)
+ if not convert_ret:
+ logger_f.error(f"convert onnx model to trt failed")
+ exit(1)
+
+ infer_cfg.infer_params.flag_pasteback = True
+ pipe = FasterLivePortraitPipeline(cfg=infer_cfg, is_animal=True)
+
+
+def run_with_video(source_image_path, driving_video_path, save_dir):
+ global pipe
+ ret = pipe.prepare_source(source_image_path, realtime=False)
+ if not ret:
+ logger_f.warning(f"no face in {source_image_path}! exit!")
+ return
+ vcap = cv2.VideoCapture(driving_video_path)
+ fps = int(vcap.get(cv2.CAP_PROP_FPS))
+ h, w = pipe.src_imgs[0].shape[:2]
+
+ # render output video
+ fourcc = cv2.VideoWriter_fourcc(*'mp4v')
+ vsave_crop_path = os.path.join(save_dir,
+ f"{os.path.basename(source_image_path)}-{os.path.basename(driving_video_path)}-crop.mp4")
+ vout_crop = cv2.VideoWriter(vsave_crop_path, fourcc, fps, (512 * 2, 512))
+ vsave_org_path = os.path.join(save_dir,
+ f"{os.path.basename(source_image_path)}-{os.path.basename(driving_video_path)}-org.mp4")
+ vout_org = cv2.VideoWriter(vsave_org_path, fourcc, fps, (w, h))
+
+ infer_times = []
+ motion_lst = []
+ c_eyes_lst = []
+ c_lip_lst = []
+
+ frame_ind = 0
+ while vcap.isOpened():
+ ret, frame = vcap.read()
+ if not ret:
+ break
+ t0 = time.time()
+ first_frame = frame_ind == 0
+ dri_crop, out_crop, out_org, dri_motion_info = pipe.run(frame, pipe.src_imgs[0], pipe.src_infos[0],
+ first_frame=first_frame)
+ frame_ind += 1
+ if out_crop is None:
+ logger_f.warning(f"no face in driving frame:{frame_ind}")
+ continue
+
+ motion_lst.append(dri_motion_info[0])
+ c_eyes_lst.append(dri_motion_info[1])
+ c_lip_lst.append(dri_motion_info[2])
+
+ infer_times.append(time.time() - t0)
+ # print(time.time() - t0)
+ dri_crop = cv2.resize(dri_crop, (512, 512))
+ out_crop = np.concatenate([dri_crop, out_crop], axis=1)
+ out_crop = cv2.cvtColor(out_crop, cv2.COLOR_RGB2BGR)
+ vout_crop.write(out_crop)
+ out_org = cv2.cvtColor(out_org, cv2.COLOR_RGB2BGR)
+ vout_org.write(out_org)
+ vcap.release()
+ vout_crop.release()
+ vout_org.release()
+ if video_has_audio(driving_video_path):
+ vsave_crop_path_new = os.path.splitext(vsave_crop_path)[0] + "-audio.mp4"
+ subprocess.call(
+ [FFMPEG, "-i", vsave_crop_path, "-i", driving_video_path,
+ "-b:v", "10M", "-c:v",
+ "libx264", "-map", "0:v", "-map", "1:a",
+ "-c:a", "aac",
+ "-pix_fmt", "yuv420p", vsave_crop_path_new, "-y", "-shortest"])
+ vsave_org_path_new = os.path.splitext(vsave_org_path)[0] + "-audio.mp4"
+ subprocess.call(
+ [FFMPEG, "-i", vsave_org_path, "-i", driving_video_path,
+ "-b:v", "10M", "-c:v",
+ "libx264", "-map", "0:v", "-map", "1:a",
+ "-c:a", "aac",
+ "-pix_fmt", "yuv420p", vsave_org_path_new, "-y", "-shortest"])
+
+ logger_f.info(vsave_crop_path_new)
+ logger_f.info(vsave_org_path_new)
+ else:
+ logger_f.info(vsave_crop_path)
+ logger_f.info(vsave_org_path)
+
+ logger_f.info(
+ "inference median time: {} ms/frame, mean time: {} ms/frame".format(np.median(infer_times) * 1000,
+ np.mean(infer_times) * 1000))
+ # save driving motion to pkl
+ template_dct = {
+ 'n_frames': len(motion_lst),
+ 'output_fps': fps,
+ 'motion': motion_lst,
+ 'c_eyes_lst': c_eyes_lst,
+ 'c_lip_lst': c_lip_lst,
+ }
+ template_pkl_path = os.path.join(save_dir,
+ f"{os.path.basename(driving_video_path)}.pkl")
+ with open(template_pkl_path, "wb") as fw:
+ pickle.dump(template_dct, fw)
+ logger_f.info(f"save driving motion pkl file at : {template_pkl_path}")
+
+
+def run_with_pkl(source_image_path, driving_pickle_path, save_dir):
+ global pipe
+ ret = pipe.prepare_source(source_image_path, realtime=False)
+ if not ret:
+ logger_f.warning(f"no face in {source_image_path}! exit!")
+ return
+
+ with open(driving_pickle_path, "rb") as fin:
+ dri_motion_infos = pickle.load(fin)
+
+ fps = int(dri_motion_infos["output_fps"])
+ h, w = pipe.src_imgs[0].shape[:2]
+
+ # render output video
+ fourcc = cv2.VideoWriter_fourcc(*'mp4v')
+ vsave_crop_path = os.path.join(save_dir,
+ f"{os.path.basename(source_image_path)}-{os.path.basename(driving_pickle_path)}-crop.mp4")
+ vout_crop = cv2.VideoWriter(vsave_crop_path, fourcc, fps, (512, 512))
+ vsave_org_path = os.path.join(save_dir,
+ f"{os.path.basename(source_image_path)}-{os.path.basename(driving_pickle_path)}-org.mp4")
+ vout_org = cv2.VideoWriter(vsave_org_path, fourcc, fps, (w, h))
+
+ infer_times = []
+ motion_lst = dri_motion_infos["motion"]
+ c_eyes_lst = dri_motion_infos["c_eyes_lst"] if "c_eyes_lst" in dri_motion_infos else dri_motion_infos[
+ "c_d_eyes_lst"]
+ c_lip_lst = dri_motion_infos["c_lip_lst"] if "c_lip_lst" in dri_motion_infos else dri_motion_infos["c_d_lip_lst"]
+
+ frame_num = len(motion_lst)
+ for frame_ind in tqdm(range(frame_num)):
+ t0 = time.time()
+ first_frame = frame_ind == 0
+ dri_motion_info_ = [motion_lst[frame_ind], c_eyes_lst[frame_ind], c_lip_lst[frame_ind]]
+ out_crop, out_org = pipe.run_with_pkl(dri_motion_info_, pipe.src_imgs[0], pipe.src_infos[0],
+ first_frame=first_frame)
+ if out_crop is None:
+ logger_f.warning(f"no face in driving frame:{frame_ind}")
+ continue
+
+ infer_times.append(time.time() - t0)
+ # print(time.time() - t0)
+ out_crop = cv2.cvtColor(out_crop, cv2.COLOR_RGB2BGR)
+ vout_crop.write(out_crop)
+ out_org = cv2.cvtColor(out_org, cv2.COLOR_RGB2BGR)
+ vout_org.write(out_org)
+
+ vout_crop.release()
+ vout_org.release()
+ logger_f.info(vsave_crop_path)
+ logger_f.info(vsave_org_path)
+ logger_f.info(
+ "inference median time: {} ms/frame, mean time: {} ms/frame".format(np.median(infer_times) * 1000,
+ np.mean(infer_times) * 1000))
+
+
+class LivePortraitParams(BaseModel):
+ flag_pickle: bool = False
+ flag_relative_input: bool = True
+ flag_do_crop_input: bool = True
+ flag_remap_input: bool = True
+ driving_multiplier: float = 1.0
+ flag_stitching: bool = True
+ flag_crop_driving_video_input: bool = True
+ flag_video_editing_head_rotation: bool = False
+ flag_is_animal: bool = True
+ scale: float = 2.3
+ vx_ratio: float = 0.0
+ vy_ratio: float = -0.125
+ scale_crop_driving_video: float = 2.2
+ vx_ratio_crop_driving_video: float = 0.0
+ vy_ratio_crop_driving_video: float = -0.1
+ driving_smooth_observation_variance: float = 1e-7
+
+
+@app.post("/predict/")
+async def upload_files(
+ source_image: Optional[UploadFile] = File(None),
+ driving_video: Optional[UploadFile] = File(None),
+ driving_pickle: Optional[UploadFile] = File(None),
+ flag_is_animal: bool = Form(...),
+ flag_pickle: bool = Form(...),
+ flag_relative_input: bool = Form(...),
+ flag_do_crop_input: bool = Form(...),
+ flag_remap_input: bool = Form(...),
+ driving_multiplier: float = Form(...),
+ flag_stitching: bool = Form(...),
+ flag_crop_driving_video_input: bool = Form(...),
+ flag_video_editing_head_rotation: bool = Form(...),
+ scale: float = Form(...),
+ vx_ratio: float = Form(...),
+ vy_ratio: float = Form(...),
+ scale_crop_driving_video: float = Form(...),
+ vx_ratio_crop_driving_video: float = Form(...),
+ vy_ratio_crop_driving_video: float = Form(...),
+ driving_smooth_observation_variance: float = Form(...)
+):
+ # 根据传入的表单参数构建 infer_params
+ infer_params = LivePortraitParams(
+ flag_is_animal=flag_is_animal,
+ flag_pickle=flag_pickle,
+ flag_relative_input=flag_relative_input,
+ flag_do_crop_input=flag_do_crop_input,
+ flag_remap_input=flag_remap_input,
+ driving_multiplier=driving_multiplier,
+ flag_stitching=flag_stitching,
+ flag_crop_driving_video_input=flag_crop_driving_video_input,
+ flag_video_editing_head_rotation=flag_video_editing_head_rotation,
+ scale=scale,
+ vx_ratio=vx_ratio,
+ vy_ratio=vy_ratio,
+ scale_crop_driving_video=scale_crop_driving_video,
+ vx_ratio_crop_driving_video=vx_ratio_crop_driving_video,
+ vy_ratio_crop_driving_video=vy_ratio_crop_driving_video,
+ driving_smooth_observation_variance=driving_smooth_observation_variance
+ )
+
+ global pipe
+ pipe.init_vars()
+ if infer_params.flag_is_animal != pipe.is_animal:
+ pipe.init_models(is_animal=infer_params.flag_is_animal)
+
+ args_user = {
+ 'flag_relative_motion': infer_params.flag_relative_input,
+ 'flag_do_crop': infer_params.flag_do_crop_input,
+ 'flag_pasteback': infer_params.flag_remap_input,
+ 'driving_multiplier': infer_params.driving_multiplier,
+ 'flag_stitching': infer_params.flag_stitching,
+ 'flag_crop_driving_video': infer_params.flag_crop_driving_video_input,
+ 'flag_video_editing_head_rotation': infer_params.flag_video_editing_head_rotation,
+ 'src_scale': infer_params.scale,
+ 'src_vx_ratio': infer_params.vx_ratio,
+ 'src_vy_ratio': infer_params.vy_ratio,
+ 'dri_scale': infer_params.scale_crop_driving_video,
+ 'dri_vx_ratio': infer_params.vx_ratio_crop_driving_video,
+ 'dri_vy_ratio': infer_params.vy_ratio_crop_driving_video,
+ }
+ # update config from user input
+ update_ret = pipe.update_cfg(args_user)
+
+ # 保存 source_image 到指定目录
+ temp_dir = os.path.join(result_dir, f"temp-{datetime.datetime.now().strftime('%Y-%m-%d-%H%M%S')}")
+ os.makedirs(temp_dir, exist_ok=True)
+ if source_image and source_image.filename:
+ source_image_path = os.path.join(temp_dir, source_image.filename)
+ with open(source_image_path, "wb") as buffer:
+ buffer.write(await source_image.read()) # 将内容写入文件
+ else:
+ source_image_path = None
+
+ if driving_video and driving_video.filename:
+ driving_video_path = os.path.join(temp_dir, driving_video.filename)
+ with open(driving_video_path, "wb") as buffer:
+ buffer.write(await driving_video.read()) # 将内容写入文件
+ else:
+ driving_video_path = None
+
+ if driving_pickle and driving_pickle.filename:
+ driving_pickle_path = os.path.join(temp_dir, driving_pickle.filename)
+ with open(driving_pickle_path, "wb") as buffer:
+ buffer.write(await driving_pickle.read()) # 将内容写入文件
+ else:
+ driving_pickle_path = None
+
+ save_dir = os.path.join(result_dir, f"{datetime.datetime.now().strftime('%Y-%m-%d-%H%M%S')}")
+ os.makedirs(save_dir, exist_ok=True)
+
+ if infer_params.flag_pickle:
+ if source_image_path and driving_pickle_path:
+ run_with_pkl(source_image_path, driving_pickle_path, save_dir)
+ else:
+ if source_image_path and driving_video_path:
+ run_with_video(source_image_path, driving_video_path, save_dir)
+ # zip all files and return
+ # 使用 BytesIO 在内存中创建一个字节流
+ zip_buffer = io.BytesIO()
+
+ # 使用 ZipFile 将文件夹内容压缩到 zip_buffer 中
+ with ZipFile(zip_buffer, "w") as zip_file:
+ for root, dirs, files in os.walk(save_dir):
+ for file in files:
+ file_path = os.path.join(root, file)
+ # 添加文件到 ZIP 文件中
+ zip_file.write(file_path, arcname=os.path.relpath(file_path, save_dir))
+
+ # 确保缓冲区指针在开始位置，以便读取整个内容
+ zip_buffer.seek(0)
+ shutil.rmtree(temp_dir)
+ shutil.rmtree(save_dir)
+ # 通过 StreamingResponse 返回 zip 文件
+ return StreamingResponse(zip_buffer, media_type="application/zip",
+ headers={"Content-Disposition": "attachment; filename=output.zip"})
+
+
+if __name__ == "__main__":
+ import uvicorn
+
+ uvicorn.run(app, host=os.environ.get("FLIP_IP", "127.0.0.1"), port=os.environ.get("FLIP_PORT", 9871))
diff --git a/assets/.gitignore b/assets/.gitignore
new file mode 100644
index 0000000000000000000000000000000000000000..892dfa4274b60c1629e26719bbd1e462fcce33e8
--- /dev/null
+++ b/assets/.gitignore
@@ -0,0 +1,2 @@
+examples/driving/*.pkl
+examples/driving/*_crop.mp4
diff --git a/assets/docs/API.md b/assets/docs/API.md
new file mode 100644
index 0000000000000000000000000000000000000000..2e4238b0d57438b37ae73131da3241645074be51
--- /dev/null
+++ b/assets/docs/API.md
@@ -0,0 +1,41 @@
+## FasterLivePortrait API Usage Guide
+
+### Building the Image
+* Decide on an image name, for example `shaoguo/faster_liveportrait_api:v1.0`. Replace the `-t` parameter in the following command with your chosen name.
+* Run `docker build -t shaoguo/faster_liveportrait_api:v1.0 -f DockerfileAPI .`
+
+### Running the Image
+Ensure that your machine has Nvidia GPU drivers installed. CUDA version should be 12.0 or higher. Two scenarios are described below.
+
+* Running on a Local Machine (typically for self-testing)
+ * Modify the image name according to what you defined above.
+ * Confirm the service port number, default is `9871`. You can define your own by changing the `SERVER_PORT` environment variable in the command below. Remember to also change `-p 9871:9871` to map the port.
+ * Set the model path environment variable `CHECKPOINT_DIR`. If you've previously downloaded FasterLivePortrait's onnx model and converted it to trt, I recommend mapping the model files into the container using `-v`, for example `-v E:\my_projects\FasterLivePortrait\checkpoints:/root/FasterLivePortrait/checkpoints`. This avoids re-downloading the onnx model and doing trt conversion. Otherwise, I will check if `CHECKPOINT_DIR` has models, and if not, I will automatically download (ensure network connectivity) and do trt conversion, which will take considerable time.
+ * Run command (note: modify the following command according to your settings):
+ ```shell
+ docker run -d --gpus=all \
+ --name faster_liveportrait_api \
+ -v E:\my_projects\FasterLivePortrait\checkpoints:/root/FasterLivePortrait/checkpoints \
+ -e CHECKPOINT_DIR=/root/FasterLivePortrait/checkpoints \
+ -e SERVER_PORT=9871 \
+ -p 9871:9871 \
+ --restart=always \
+ shaoguo/faster_liveportrait_api:v1.0 \
+ /bin/bash
+ ```
+ * Normal operation should display the following information(docker logs $container_id). The running logs are saved in `/root/FasterLivePortrait/logs/log_run.log`:
+ ```shell
+ INFO: Application startup complete.
+ INFO: Uvicorn running on http://0.0.0.0:9871 (Press CTRL+C to quit)
+ ```
+
+* Running on Cloud GPU Cluster (production environment)
+ * This needs to be configured according to different clusters, but the core is the configuration of docker image and environment variables.
+ * Load balancing may need to be set up.
+
+### API Call Testing
+Refer to `tests/test_api.py`. The default is the Animal model, but now it also supports the Human model.
+The return is a compressed package, by default unzipped to `./results/api_*`. Confirm according to the actual printed log.
+* `test_with_video_animal()`, image and video driving. Set `flag_pickle=False`. It will additionally return the driving video's pkl file, which can be called directly next time.
+* `test_with_pkl_animal()`, image and pkl driving.
+* `test_with_video_human()`, image and video driving under the Human model, set `flag_is_animal=False`
\ No newline at end of file
diff --git a/assets/docs/API_ZH.md b/assets/docs/API_ZH.md
new file mode 100644
index 0000000000000000000000000000000000000000..19e24a9dc9d138950116c8d907c3207f3af43b4b
--- /dev/null
+++ b/assets/docs/API_ZH.md
@@ -0,0 +1,47 @@
+## FasterLivePortrait API使用教程
+
+### 构建镜像
+
+* 确定镜像的名字，比如 `shaoguo/faster_liveportrait_api:v1.0`。确认后替换为下面命令 `-t` 的参数。
+* 运行 `docker build -t shaoguo/faster_liveportrait_api:v1.0 -f DockerfileAPI .`
+
+### 运行镜像
+
+请确保你的机器已经装了Nvidia显卡的驱动。CUDA的版本在cuda12.0及以上。以下分两种情况介绍。
+
+* 本地机器运行(一般自己测试使用)
+ * 镜像名称根据上面你自己定义的更改。
+ * 确认服务的端口号，默认为`9871`，你可以自己定义，更改下面命令里环境变量`SERVER_PORT`。同时要记得更改`-p 9871:9871`,
+ 将端口映射出来。
+ * 设置模型路径环境变量 `CHECKPOINT_DIR`。如果你之前下载过FasterLivePortrait的onnx模型并做过trt的转换，我建议
+ 是可以通过 `-v`把
+ 模型文件映射进入容器，比如 `-v E:\my_projects\FasterLivePortrait\checkpoints:/root/FasterLivePortrait/checkpoints`,
+ 这样就避免重新下载onnx模型和做trt的转换。否则我将会检测`CHECKPOINT_DIR`是否有模型，没有的话，我将自动下载（确保有网络）和做trt的转换，这将耗时比较久的时间。
+ * 运行命令(注意你要根据自己的设置更改以下命令的信息）：
+ ```shell
+ docker run -d --gpus=all \
+ --name faster_liveportrait_api \
+ -v E:\my_projects\FasterLivePortrait\checkpoints:/root/FasterLivePortrait/checkpoints \
+ -e CHECKPOINT_DIR=/root/FasterLivePortrait/checkpoints \
+ -e SERVER_PORT=9871 \
+ -p 9871:9871 \
+ --restart=always \
+ shaoguo/faster_liveportrait_api:v1.0
+ ```
+ * 正常运行应该会显示以下信息(docker logs container_id), 运行的日志保存在`/root/FasterLivePortrait/logs/log_run.log`:
+ ```shell
+ INFO: Application startup complete.
+ INFO: Uvicorn running on http://0.0.0.0:9871 (Press CTRL+C to quit)
+ ```
+* 云端GPU集群运行（生产环境）
+ * 这需要根据不同的集群做配置，但核心就是镜像和环境变量的配置。
+ * 可能要设置负载均衡。
+
+### API调用测试
+
+可以参考`tests/test_api.py`, 默认是Animal的模型，但现在同时也支持Human的模型了。
+返回的是压缩包，默认解压在`./results/api_*`, 根据实际打印出来的日志确认。
+
+* `test_with_video_animal()`, 图像和视频的驱动。设置`flag_pickle=False`。会额外返回driving video的pkl文件，下次可以直接调用。
+* `test_with_pkl_animal()`, 图像和pkl的驱动。
+* `test_with_video_human()`, Human模型下图像和视频的驱动，设置`flag_is_animal=False`
\ No newline at end of file
diff --git a/assets/gradio/gradio_description_animate_clear.md b/assets/gradio/gradio_description_animate_clear.md
new file mode 100644
index 0000000000000000000000000000000000000000..96d5fee236a75a418911512eaefd2b830ff03acf
--- /dev/null
+++ b/assets/gradio/gradio_description_animate_clear.md
@@ -0,0 +1,6 @@
+
+ Step 3: Click the 🚀 Animate button below to generate, or click 🧹 Clear to erase the results
+
+
diff --git a/assets/gradio/gradio_description_animation.md b/assets/gradio/gradio_description_animation.md
new file mode 100644
index 0000000000000000000000000000000000000000..126c4ce710212159279160aaf6c789315293c0d3
--- /dev/null
+++ b/assets/gradio/gradio_description_animation.md
@@ -0,0 +1,19 @@
+🔥 To animate the source image or video with the driving video, please follow these steps:
+
+1. In the Animation Options for Source Image or Video section, we recommend enabling the do crop (source) option if faces occupy a small portion of your source image or video.
+
+
+2. In the Animation Options for Driving Video section, the relative head rotation and smooth strength options only take effect if the source input is a video.
+
+
+3. Press the 🚀 Animate button and wait for a moment. Your animated video will appear in the result block. This may take a few moments. If the input is a source video, the length of the animated video is the minimum of the length of the source video and the driving video.
+
+
+4. If you want to upload your own driving video, the best practice:
+
+ - Crop it to a 1:1 aspect ratio (e.g., 512x512 or 256x256 pixels), or enable auto-driving by checking `do crop (driving video)`.
+ - Focus on the head area, similar to the example videos.
+ - Minimize shoulder movement.
+ - Make sure the first frame of driving video is a frontal face with **neutral expression**.
+
+
diff --git a/assets/gradio/gradio_description_retargeting.md b/assets/gradio/gradio_description_retargeting.md
new file mode 100644
index 0000000000000000000000000000000000000000..64f1a7c1a791e33bbe39de03d910180edfb1b795
--- /dev/null
+++ b/assets/gradio/gradio_description_retargeting.md
@@ -0,0 +1,14 @@
+
+
+
+
+
+
+
+
+
Retargeting
+
Upload a Source Portrait as Retargeting Input, then drag the sliders and click the 🚗 Retargeting button. You can try running it multiple times.
+
+ 😊 Set both ratios to 0.8 to see what's going on!
+
+
+
+
+
+ Step 1: Upload a Source Image or Video (any aspect ratio) ⬇️
+
+
+
+
+ Step 2: Upload a Driving Video (any aspect ratio) ⬇️
+
+
+ Tips: Focus on the head, minimize shoulder movement, neutral expression in first frame.
+
+
+
+
+
FasterLivePortrait: Bring Portraits to Life in Real Time
+
Built on LivePortrait
+
+
+
+ 
+ 
+