PolyEdit real-hard RL, five seeds
This public repository preserves the models used for the PolyEdit real-hard Egc results in the 2026 GPU project interim report.
Contents
kl_reinforce/checkpoints: final KL-regularized REINFORCE policy weights from seeds 1004 to 1008. These legacy runs saved only the policy state dict.grpo/checkpoints: complete SFT anchor, GRPO policy and fine-tuned Egc verifier bundles from the same five seeds.kl_reinforce/results and grpo/results: per-seed evaluation JSON files.summaries: aggregate tables and analysis.load_polyedit_bundle.py: minimal loader for complete policy and verifier bundles.
The complete GRPO bundles also contain the feature normalization statistics required to reload the policy. The legacy KL-REINFORCE weights use the same deterministic SFT feature pipeline, but their original files did not preserve these statistics or verifier parameters. They are retained as historical weights and should be interpreted with their per-seed result JSON.
Training and evaluation
- Data: frozen real DFT labels in the PolyEdit repository
- Property: chain bandgap,
Egc
- Training requests: 369
- Evaluation: 112 single-step and 36 multi-step requests
- Seeds: 1004, 1005, 1006, 1007, 1008
- Final evaluator: frozen DFT label lookup, separate from the learned reward verifier
- SFT: 60 epochs with set-valued trajectory supervision
- KL-REINFORCE: 500 iterations, batch 32, KL coefficient 0.5
- GRPO: 250 iterations, request batch 8, group size 8, two update epochs, clip 0.2, KL coefficient 0.5
- Reward: verifier target hit + normalized target-distance improvement - 0.05 per action
Mean results
Values are five-seed mean with 95% t confidence interval.
| Method | Single success | Multi success |
|---|
| SFT | 0.887 ± 0.030 | 0.800 ± 0.082 |
| KL-REINFORCE | 0.870 ± 0.028 | 0.772 ± 0.057 |
| GRPO | 0.818 ± 0.028 | 0.672 ± 0.110 |
GRPO did not improve over SFT or KL-REINFORCE under this reward. This negative result is included to avoid checkpoint or method selection bias.
External models referenced by the report
The interim report also references public upstream checkpoints. They are not duplicated here.
HAYDERphd/polyBERT, revision 7bf9ed32ac54dea5bc163cf90100728b49341750luca0621/mmpae-1p5B-e260-2gpu-bs128-lr5e5-b2000, revision 20d49c79c7939eac830d1d919f97086e1833079bpromotion/mmpae-3B-e300-2xb200-bs128-lr5e5-b1500, revision d963559e8d95b227c46502d6f1972a696796067fnamhokaist/mmpae-8b-e200-ddp4, revision 4af91617ff1b00bbd46370af4fd391e3f6826f59
Limitations
The action space is restricted to a within-dataset matched-pair graph. The reward verifier is trained on limited DFT labels and can be exploited. GRPO results are specific to the stated group size, reward and KL setting. The 36-request multi-step interval is wide, so the result should not be generalized beyond this benchmark.
Code
