arxiv:2503.10291

VisualPRM: An Effective Process Reward Model for Multimodal Reasoning

Published on Mar 13

· Submitted by

Weiyun1025 on Mar 14

Upvote

Authors:

Weiyun Wang ,

Lianjie Chen ,

Xiangyu Zhao ,

Shenglong Ye ,

Xizhou Zhu ,

Lewei Lu ,

Haodong Duan ,

Jifeng Dai ,

Wenhai Wang

Abstract

We introduce VisualPRM, an advanced multimodal Process Reward Model (PRM) with 8B parameters, which improves the reasoning abilities of existing Multimodal Large Language Models (MLLMs) across different model scales and families with Best-of-N (BoN) evaluation strategies. Specifically, our model improves the reasoning performance of three types of MLLMs and four different model scales. Even when applied to the highly capable InternVL2.5-78B, it achieves a 5.9-point improvement across seven multimodal reasoning benchmarks. Experimental results show that our model exhibits superior performance compared to Outcome Reward Models and Self-Consistency during BoN evaluation. To facilitate the training of multimodal PRMs, we construct a multimodal process supervision dataset VisualPRM400K using an automated data pipeline. For the evaluation of multimodal PRMs, we propose VisualProcessBench, a benchmark with human-annotated step-wise correctness labels, to measure the abilities of PRMs to detect erroneous steps in multimodal reasoning tasks. We hope that our work can inspire more future research and contribute to the development of MLLMs. Our model, data, and benchmark are released in https://internvl.github.io/blog/2025-03-13-VisualPRM/.

View arXiv page View PDF Project page GitHub repository Add to collection

Community

Weiyun1025

Paper author Paper submitter about 12 hours ago

The main contributions of this paper are as follows:

VisualPRM-8B: an advanced multimodal Process Reward Model (PRM) with 8B parameters. Specifically, VisualPRM improves the overall reasoning performance of MiniCPM-V2.6, QwenVL2.5-7B, InternVL2.5-8B, and InternVL2.5-78B by 8.0, 3.7, 8.4, and 5.9 points, respectively, across seven multimodal reasoning benchmarks. Additionally, we compare PRMs with Outcome Reward Models and Self-Consistency in BoN evaluation, finding that PRMs consistently outperform both approaches.
VisualPRM400K: a dataset comprising approximately 400K multimodal process supervision data. We generate the data using an automatic data pipeline. The key idea is to estimate the expected accuracy $m c_{i}$ of the given step $s_{\leq i}$ based on Monte Carlo sampling and consider the step correct if $m c_{i} > 0$ .
VisualProcessBench: a benchmark designed to measure the abilities of PRMs and MLLMs to identify erroneous steps in multimodal reasoning tasks. This benchmark comprises 2,866 samples with a total of 26,950 human-annotated step-wise correctness labels.