You can deploy this model and start earning money today!

yinita/
ps4mas-grpo-9b-sonnet-large-step200

Name: ps4mas-grpo-9b-sonnet-large-step200
Availability: InStock

Views

No views yet

PS4MAS GRPO LoRA — Qwen3.5-9B (Sonnet judge, 4 cold topologies)

LoRA adapter from grpo_9b_sonnet_large_200_0622_0511 (200-step GRPO, Bedrock Claude Sonnet 4.6 judger).

Checkpoint note

Best in-run eval was step 175 (reward -0.875, composite ~2.13), but only save_steps=50 checkpoints were kept (keep_checkpoints=1). This repo contains step 200 — the only surviving numbered checkpoint.

Step	reward	~composite	saved?
175	-0.875	2.13	no (eval peak)
200	-1.009	1.99	yes (this repo)

Training config

Topologies: PS-cold-single, PS-cold-central, PS-cold-hier, PS-cold-debate
LoRA r=16, alpha=32, target_modules: q/k/v/o_proj, gate/up/down_proj
group_size=8, groups_per_step=8, temperature=0.8
Judge: us.anthropic.claude-sonnet-4-6 (Bedrock)

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen3.5-9B"
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "yinita/ps4mas-grpo-9b-sonnet-large-step200")
tokenizer = AutoTokenizer.from_pretrained("yinita/ps4mas-grpo-9b-sonnet-large-step200")

For vLLM LoRA: load base Qwen3.5-9B + this adapter (see PS4MAS eval scripts).

W&B

https://wandb.ai/yinita/ps4mas-drmas/runs/5915a52fba046f09