Visual-ERM
Visual-ERM is a multimodal generative reward model for vision-to-code tasks.
It evaluates outputs directly in the rendered visual space and produces fine-grained, interpretable, and task-agnostic discrepancy feedback for structured visual reconstruction.
Model Overview
Existing rewards for vision-to-code usually fall into two categories:
- Text-based rewards such as edit distance or TEDS, which ignore important visual cues like layout, spacing, alignment, and style.
- Vision embedding rewards such as DINO similarity, which are often coarse-grained and can be vulnerable to reward hacking.
Visual-ERM addresses this by directly comparing:
- the ground-truth image, and
- the rendered image produced from a model prediction,
and then generating structured discrepancy annotations that can be converted into reward signals or used for reflection-based refinement.
What this model does
Visual-ERM is designed to judge whether a predicted result is visually equivalent to the target.
Given a pair of images, it can identify discrepancies such as:
- category
- severity
- location
- description
This makes Visual-ERM useful not only as a reward model for RL, but also as a visual critic for test-time reflection and revision.
Supported Tasks
Visual-ERM is designed for structured visual reconstruction tasks, including:
- Chart-to-Code
- Table-to-Markdown
- SVG-to-Code
Key Features
-
Visual-space reward modeling
Evaluates predictions in rendered visual space instead of relying only on text matching or coarse embedding similarity.
-
Fine-grained and interpretable feedback
Produces structured discrepancy annotations rather than a single black-box score.
-
Task-agnostic reward supervision
A unified reward model that generalizes across multiple vision-to-code tasks.
-
Useful for both training and inference
Can be used as a reward model in RL and as a visual critic during test-time refinement.
VC-RewardBench
We also release VisualCritic-RewardBench (VC-RewardBench), a benchmark for evaluating fine-grained image-to-image discrepancy judgment on structured visual data.
Benchmark Features
- Covers charts, tables, and SVGs
- Contains 1,335 carefully curated instances
- Each instance includes:
- a ground-truth image
- a corrupted / rendered counterpart
- fine-grained discrepancy annotations
How to Use
Visual-ERM is fine-tuned from Qwen/Qwen3-VL-8B-Instruct and follows the same multimodal interface.
Input
Visual-ERM takes as input:
- a reference / ground-truth image
- a rendered prediction image
- a prompt asking the model to identify fine-grained visual discrepancies
Output
The model outputs structured discrepancy annotations, which can then be:
- converted into a scalar reward for RL
- used as feedback for reflection-and-revision
- evaluated directly on VC-RewardBench
A typical output format is:
1{
2 "errors": [
3 {
4 "category": "structure_error",
5 "severity": 3,
6 "location": "legend area",
7 "description": "The legend is placed outside the plot area in the prediction."
8 },
9 {
10 "category": "style_error",
11 "severity": 2,
12 "location": "bar colors",
13 "description": "The colors differ from those in the reference image."
14 }
15 ]
16}
Inference / Evaluation / RL
For full inference scripts, RL training pipelines, evaluation code, and prompt templates, please refer to the official repository:
Intended Use
Visual-ERM is intended for:
- reward modeling in vision-to-code RL pipelines
- visual discrepancy judgment between target and predicted renderings
- reflection-based refinement at inference time
- research on visual reward modeling and multimodal RL
Citation
If you find this model useful, please consider citing:
1@article{liu2026visual,
2 title={Visual-ERM: Reward Modeling for Visual Equivalence},
3 author={Liu, Ziyu and Ding, Shengyuan and Fang, Xinyu and Dai, Xuanlang and Yang, Penghui and Liang, Jianze and Wang, Jiaqi and Chen, Kai and Lin, Dahua and Zang, Yuhang},
4 journal={arXiv preprint arXiv:2603.13224},
5 year={2026}
6}
Contact
If you are interested in visual reward modeling, vision-to-code, or reinforcement learning for multimodal models, feel free to reach out.
