Update README.md

README.md

@@ -1,3 +1,125 @@
----
-license: agpl-3.0
----
+---
+license: agpl-3.0
+language:
+- en
+base_model: microsoft/Florence-2-base-ft
+tags:
+- rknn
+---
+
+# Florence-2-base-ft-ONNX-RKNN2
+
+### (English README see below)
+
+ONNX/RKNN2部署Florence-2视觉多模态大模型! 
+
+- 推理速度(RKNN2)：RK3588推理一张512x512图片, 使用`<MORE_DETAILED_CAPTION>`指令, 总时间需要~2秒
+- 内存占用(RKNN2)：约1.9GB
+
+## 使用方法
+
+1. 克隆项目到本地
+
+2. 安装依赖
+
+```bash
+pip install transformers onnxruntime pillow numpy<2
+```
+
+如果需要使用rknn推理, 还需要安装rknn-toolkit2-lite2. 
+
+3. 修改项目路径
+分词器和预处理配置仍然需要使用原项目中的文件. 将onnx/onnxrun.py或onnx/rknnrun.py中的对应路径修改为项目所在路径. 
+```python
+AutoProcessor.from_pretrained(
+    "path/to/Florence-2-base-ft-ONNX-RKNN2",
+    trust_remote_code=True
+)
+```
+
+4. 运行
+```bash
+python onnx/onnxrun.py # 或 python onnx/rknnrun.py
+```
+
+## RKNN模型转换
+
+你需要提前安装rknn-toolkit2 v2.1.0或更高版本. 
+
+```bash
+cd onnx
+python convert.py all
+```
+
+注意: RKNN模型不支持运行时任意长度的输入, 所以需要提前确定好输入的shape. 之后可以修改convert.py中的`vision_size`, `vision_tokens`, `prompt_tokens`来修改输入的shape. 
+
+## 存在的问题(rknn)
+- 已知转换vision encoder时输入分辨率>=640x640时会报错`Buffer overflow!`然后转换失败, 所以目前是使用512x512的输入, 推理质量会下降
+- 在同分辨率下推理精度相比onnxruntime有显著下降, 已确定问题出在vision encoder部分. 如果对精度要求高, 这部分可以换成onnxruntime
+- vision encoder推理需要1.5秒, 占比最大, 但其中大半时间都在做Transpose, 也许还可以优化
+- decode阶段因为kvcache的长度不断变化, 貌似无法简单的使用NPU推理. 不过用onnxruntime应该也足够了
+
+## 参考
+- [microsoft/Florence-2-base-ft](https://huggingface.co/microsoft/Florence-2-base-ft)
+- [onnx-community/Florence-2-base-ft](https://huggingface.co/onnx-community/Florence-2-base-ft)
+- [florence2-webgpu](https://huggingface.co/spaces/Xenova/florence2-webgpu)
+
+
+# English README
+
+# Florence-2-base-ft-ONNX-RKNN2
+
+ONNX/RKNN2 deployment for Florence-2 visual-language multimodal large model!
+
+- Inference speed (RKNN2): RK3588 inference with a 512x512 image, using the `<MORE_DETAILED_CAPTION>` instruction, takes ~2 seconds in total.
+- Memory usage (RKNN2): Approximately 1.9GB
+
+## Usage
+
+1. Clone the project locally
+
+2. Install dependencies
+
+```bash
+pip install transformers onnxruntime pillow numpy<2
+```
+
+If you need to use rknn for inference, you also need to install rknn-toolkit2-lite2.
+
+3. Modify project paths
+The tokenizer and preprocessing configurations still need to use files from the original project. Modify the corresponding paths in onnx/onnxrun.py or onnx/rknnrun.py to the project's location.
+```python
+AutoProcessor.from_pretrained(
+    "path/to/Florence-2-base-ft-ONNX-RKNN2",
+    trust_remote_code=True
+)
+```
+
+4. Run
+```bash
+python onnx/onnxrun.py # or python onnx/rknnrun.py
+```
+
+## RKNN Model Conversion
+
+You need to install rknn-toolkit2 v2.1.0 or higher in advance.
+
+```bash
+cd onnx
+python convert.py all
+```
+
+Note: The RKNN model does not support arbitrary input lengths at runtime, so you need to determine the input shape in advance. You can modify `vision_size`, `vision_tokens`, and `prompt_tokens` in convert.py to change the input shape.
+
+## Existing Issues (rknn)
+
+- It is known that converting the vision encoder with an input resolution >= 640x640 will result in a `Buffer overflow!` error and conversion failure. Therefore, the current input is 512x512, which will reduce inference quality.
+- Inference accuracy is significantly lower compared to onnxruntime at the same resolution. The problem has been identified in the vision encoder part. If high accuracy is required, this part can be replaced with onnxruntime.
+- Vision encoder inference takes 1.5 seconds, accounting for the largest proportion, but most of this time is spent on Transpose op, which may be further optimized.
+- The decode phase seems unable to simply use NPU inference because the length of kvcache keeps changing. However, using onnxruntime should be sufficient.
+
+## References
+
+- [microsoft/Florence-2-base-ft](https://huggingface.co/microsoft/Florence-2-base-ft)
+- [onnx-community/Florence-2-base-ft](https://huggingface.co/onnx-community/Florence-2-base-ft)
+- [florence2-webgpu](https://huggingface.co/spaces/Xenova/florence2-webgpu)