LoRA fine-tune of OpenVLA-OFT on LIBERO-10 Task 9 ("put the yellow and white mug in the microwave and close it") using GRPO with dense rewards from Robo-Dopamine's GRM.

Training

Parameter	Value
Base model	RLinf/RLinf-OpenVLAOFT-LIBERO-130
Fine-tuning	LoRA via GRPO (Group Relative Policy Optimization)
Reward model	Robo-Dopamine GRM-3B (cumulative eval mode)
Task	LIBERO-10 #9: put the yellow and white mug in the microwave and close it
Hardware	4x A100-80GB (1 for GRM, 3 for FSDP training)
Epochs	20
Parallel envs	24
Infrastructure	RLinf v0.2.0

Results

The base model started at 86.5% success rate. Dense GRM-only rewards caused reward hacking — policy loss converged but success rate collapsed to 0% by epoch 20. The step-10 checkpoint (61.5% success) is the last usable checkpoint.

See TRAINING_REPORT.md in the associated dataset repo for full failure analysis.

Checkpoints

global_step_10/ — Epoch 10, 61.5% success rate (before collapse)
global_step_20/ — Epoch 20, 0% success rate (collapsed)

Limitations

This model demonstrates reward hacking from dense RL without sparse reward anchoring or KL regularization. It is released for research purposes — to study failure modes in reward-model-guided policy optimization.

Citation

bibtex

@misc{auryal2026grm-grpo,
  title={Dense Reward RL with GRM for Robotic Manipulation},
  author={Auryal},
  year={2026},
}

Auryal/openvla-oft-libero-microwave-grm-grpo

OpenVLA-OFT LIBERO Microwave (GRM+GRPO)

Training

Results

Checkpoints

Limitations

Citation

Auryal/
openvla-oft-libero-microwave-grm-grpo