Florence-2-base-ft-ONNX-RKNN2

(English README see below)

ONNX/RKNN2部署Florence-2视觉多模态大模型!

• 推理速度(RKNN2)：RK3588推理一张768x768图片, 使用<MORE_DETAILED_CAPTION>指令, 总时间需要~4.5秒
• 内存占用(RKNN2)：约2GB

使用方法

1. 克隆项目到本地

2. 安装依赖

pip install transformers onnxruntime pillow numpy<2

如果需要使用rknn推理, 还需要安装rknn-toolkit2-lite2.

3. 修改项目路径 分词器和预处理配置仍然需要使用原项目中的文件. 将onnx/onnxrun.py或onnx/rknnrun.py中的对应路径修改为项目所在路径.

AutoProcessor.from_pretrained(
    "path/to/Florence-2-base-ft-ONNX-RKNN2",
    trust_remote_code=True
)

4. 运行

python onnx/onnxrun.py # 或 python onnx/rknnrun.py

RKNN模型转换

你需要提前安装rknn-toolkit2 v2.1.0或更高版本.

cd onnx
python convert.py all

注意: RKNN模型不支持运行时任意长度的输入, 所以需要提前确定好输入的shape. 之后可以修改convert.py中的vision_size, vision_tokens, prompt_tokens来修改输入的shape.

存在的问题(rknn)

• 已知转换vision encoder时输入分辨率>=640x640时会报错Buffer overflow!然后转换失败, 所以目前是使用512x512的输入, 推理质量会下降 (已解决)
• 在同分辨率下推理精度相比onnxruntime有显著下降, 已确定问题出在vision encoder部分. 如果对精度要求高, 这部分可以换成onnxruntime (已解决, 不过现在貌似仍然会丢失一点精度)
• vision encoder推理需要1.5秒, 占比最大, 但其中大半时间都在做Transpose, 也许还可以优化 (已优化)
• decode阶段因为kvcache的长度不断变化, 貌似无法简单的使用NPU推理. 不过用onnxruntime应该也足够了

参考

• microsoft/Florence-2-base-ft
• onnx-community/Florence-2-base-ft
• florence2-webgpu

English README

Florence-2-base-ft-ONNX-RKNN2

ONNX/RKNN2 deployment for Florence-2 visual-language multimodal large model!

• Inference speed (RKNN2): RK3588 inference with a 768x768 image, using the <MORE_DETAILED_CAPTION> instruction, takes ~4.5 seconds in total.
• Memory usage (RKNN2): Approximately 2GB

Usage

1. Clone the project locally

2. Install dependencies

pip install transformers onnxruntime pillow numpy<2

If you need to use rknn for inference, you also need to install rknn-toolkit2-lite2.

3. Modify project paths The tokenizer and preprocessing configurations still need to use files from the original project. Modify the corresponding paths in onnx/onnxrun.py or onnx/rknnrun.py to the project's location.

AutoProcessor.from_pretrained(
    "path/to/Florence-2-base-ft-ONNX-RKNN2",
    trust_remote_code=True
)

4. Run

python onnx/onnxrun.py # or python onnx/rknnrun.py

RKNN Model Conversion

You need to install rknn-toolkit2 v2.1.0 or higher in advance.

cd onnx
python convert.py all

Note: The RKNN model does not support arbitrary input lengths at runtime, so you need to determine the input shape in advance. You can modify vision_size, vision_tokens, and prompt_tokens in convert.py to change the input shape.

Existing Issues (rknn)

• It is known that converting the vision encoder with an input resolution >= 640x640 will result in a Buffer overflow! error and conversion failure. Therefore, the current input is 512x512, which will reduce inference quality. (Solved.)
• Inference accuracy is significantly lower compared to onnxruntime at the same resolution. The problem has been identified in the vision encoder part. If high accuracy is required, this part can be replaced with onnxruntime. (Solved, although there is still some precision loss present.)
• Vision encoder inference takes 1.5 seconds, accounting for the largest proportion, but most of this time is spent on Transpose op, which may be further optimized. (Solved.)
• The decode phase seems unable to simply use NPU inference because the length of kvcache keeps changing. However, using onnxruntime should be sufficient.

References

• microsoft/Florence-2-base-ft
• onnx-community/Florence-2-base-ft
• florence2-webgpu