A 1-step continuation GRPO checkpoint of lyx02klmy/qwen3-4b-cp26-grpo-v5step1-p60n18 on a tiny, high-quality slice of circle-packing reflection trajectories sampled from v5 step-1 itself.

This is the second step of a base→step-1→step-2 training chain on the circle-packing task (n=26 circles in a unit square, maximize sum of radii). Each step trains for exactly one GRPO update on prompts whose previous-round response (the "parent") scored above 2.6 in 100-round sequential revision.

Training summary

Field	Value
Init	`lyx02klmy/qwen3-4b-cp26-grpo-v5step1-p60n18` (v5 step-1)
Algorithm	GRPO (group-relative PPO), no KL penalty, no entropy bonus
Optimizer	AdamW, lr=5e-6
Steps	1 (single update)
Train batch	14 prompts × n=8 rollouts = 112 responses
Sampling	temperature=1.0, max_response_length=32768
Reward	circle-packing scorer (`U` = sum_radii if valid, `max(sum_r − λ·violation, −1)` if near-valid, else −1; λ=1.0; gaming cap 2.64)
Hardware	2× A100-80GB, FSDP + vLLM rollout
Wall time	~38 min

Training data

14 prompts drawn from a sequential-revision (reflection) inference run of the v5 step-1 ckpt, max_rounds=100, 2 parallel runs. Filter: parent_score > 2.6 (i.e. the round being revised was already a near-best valid configuration).

Source: /workspace/post_training_step2_v5_p60n18/reflection_100/
Run 1 best: 2.6252 (5 prompts kept)
Run 2 best: 2.6255 (9 prompts kept)
Total kept at parent>2.6: 14
Parent score distribution: min 2.6012, max 2.6255, mean 2.6139

Step-1 metrics

Training rollouts (n=8 sampling per prompt, 112 total):

Metric	Value
n_valid	80 / 112 (71.4%)
score / mean	2.254
score / max	2.627
advantages range	[−2.475, +1.614]
response length / mean	11,952 tokens
response length / max	16,872 tokens

Validation (n=1 greedy, do_sample=False, on the 14 train prompts):

Metric	Value
score_mean	0.029 (vs −0.156 at v5 step-1)
score_max	2.6146
is_valid	3/14 (21.4%)
near_valid	1/14 (7.1%)

Greedy validation improved from v5-step-1 (mean −0.156 → +0.029, valid 16.7% → 21.4%), suggesting the second update sharpened the policy without collapsing diversity (training-time max held at 2.627).

Intended use

Research artifact. The reward is gamed at 2.64 by design, so this checkpoint is not suitable for any downstream "circle packing" deployment. Useful primarily for studying:

Iterated tiny-batch RL (1-step updates, 14-18 prompts per step)
High-parent-score data filtering as a self-distillation signal across iterations
The interaction between RL training and downstream sequential-revision performance

How to load

python

from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "lyx02klmy/qwen3-4b-cp26-grpo-v5step2-p60n14"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, dtype="bfloat16", device_map="auto")

Limitations

Single-task RL artifact — general-purpose capabilities may have shifted.
Greedy decoding is much weaker than temperature sampling.
No safety alignment beyond the base Qwen3-4B model.