Florence-2-base-ft-ONNX-RKNN2

(English README see below)

• 推理速度(RKNN2)：RK3588推理一张768x768图片, 使用<MORE_DETAILED_CAPTION>指令, 总时间需要~4秒
• 内存占用(RKNN2)：约2GB

使用方法

1. 克隆项目到本地(开发板上)
2. 安装依赖

1. 运行

RKNN模型转换

存在的问题(rknn)

• 已知转换vision encoder时输入分辨率>=640x640时会报错Buffer overflow!然后转换失败, 所以目前是使用512x512的输入, 推理质量会下降 (已解决)
• 在同分辨率下推理精度相比onnxruntime有显著下降, 已确定问题出在vision encoder部分. 如果对精度要求高, 这部分可以换成onnxruntime (已解决, 不过现在貌似仍然会丢失一点精度)
• vision encoder推理需要1.5秒, 占比最大, 但其中大半时间都在做Transpose, 也许还可以优化 (已优化)
• decode阶段因为kvcache的长度不断变化, 貌似无法简单的使用NPU推理. 不过用onnxruntime应该也足够了
• 理论上可以使用rkllm推理decoder, 但因为rkllm缺少固定位置编码的支持暂时无法实现。参考：https://github.com/airockchip/rknn-llm/issues/296

参考

• microsoft/Florence-2-base-ft
• onnx-community/Florence-2-base-ft
• florence2-webgpu

English README

• Inference Speed (RKNN2): On RK3588, inferring a 768x768 image with the <MORE_DETAILED_CAPTION> instruction takes ~4 seconds in total.
• Memory Usage (RKNN2): About 2GB.

Usage

1. Clone the project locally (on your development board).
2. Install dependencies:

1. Run:

RKNN Model Conversion

Known Issues (rknn)

• When converting the vision encoder, an input resolution of >=640x640 would cause a Buffer overflow! error and fail the conversion. Therefore, a 512x512 input was used, which degraded inference quality. (Resolved)
• Inference precision was significantly lower compared to onnxruntime at the same resolution. The issue was identified in the vision encoder part. For higher precision requirements, onnxruntime could be used for this part. (Resolved, though there might still be a slight loss of precision)
• Vision encoder inference took 1.5 seconds, accounting for the largest portion of time, with most of it spent on Transpose operations. There might be room for optimization. (Optimized)
• In the decode stage, due to the constantly changing length of the kvcache, it seems that NPU inference cannot be simply used. However, onnxruntime should be sufficient.
• Theoretically, rkllm could be used for decoder inference, but it's not currently feasible due to the lack of support for fixed positional encoding in rkllm. See: https://github.com/airockchip/rknn-llm/issues/296

References

• microsoft/Florence-2-base-ft
• onnx-community/Florence-2-base-ft
• florence2-webgpu