A LoRA adapter trained with GRPO (Group-Relative Policy Optimization) on top of the SFT warm-start SnehShah/house-md-sft-gemma3-4b, using episode rollouts against the live OpenEnv Space SnehShah/house-md-env.

This is the production submission for the Apr '26 Meta OpenEnv Hackathon — the model whose reward curve is published on W&B and whose eval JSON is included in the repo.

GitHub repo (training pipeline, notebooks, eval, blog): https://github.com/sneh2909/Overfitters Live env: https://huggingface.co/spaces/SnehShah/house-md-env W&B run (full reward, gradients, mid-eval): https://wandb.ai/sneh2909-christ-university/house-md?nw=nwusersneh2909 Frozen comparison evals: SnehShah/house-md-results

How it was trained


Base model	`unsloth/gemma-3-4b-it-unsloth-bnb-4bit`
Warm start	LoRA from `SnehShah/house-md-sft-gemma3-4b`
Method	TRL `GRPOTrainer` (group-relative advantage; KL regularizer to ref)
Reward fn	sum of 5 rubrics surfaced by the Space's `/step`
Group size	8 rollouts per prompt
Adapter	LoRA, r=32, alpha=64, target=all attn + MLP
Steps	~150 (see W&B for the exact step count)
Optimizer	AdamW 8-bit, LR=5e-6
Hardware	HF Jobs L4 ×1

Training script: scripts/train_grpo_optimized.py. Reproduction notebook: notebooks/03_grpo.ipynb.

Reward design

Rubric	Range	Captures
`r1_accuracy`	-2 – +1	Right disease and saw the necessary evidence
`r2_cost`	-1.5 – +1	Sweet-spot $200–500; large penalties at $1500+
`r6_anchoring`	-0.5 – +0.6	Did the agent revise its differential meaningfully
`r7_safety`	-2 – 0	Penalty-only; severity-scaled wrong dx / timeouts
`r8_format`	0 – 1	Fraction of valid (in-vocab, well-formed) actions

Default composite weights: {r1: 2.0, r2: 0.5, r6: 0.3, r7: 1.0, r8: 0.5}.

Use

python

from unsloth import FastLanguageModel

model, tok = FastLanguageModel.from_pretrained(
    model_name = "unsloth/gemma-3-4b-it-unsloth-bnb-4bit",
    max_seq_length = 4096,
    load_in_4bit = True,
)
model.load_adapter("SnehShah/house-md-grpo-optimized-gemma3-4b-v3")
FastLanguageModel.for_inference(model)

from house_md_env import HouseMDEnv, HouseMDAction
with HouseMDEnv(base_url="https://snehshah-house-md-env.hf.space") as env:
    res = env.reset(seed=0)
    # ... build prompt from res.observation, generate, parse JSON action,
    #     env.step(...) until res.observation.terminal

The exact eval harness is scripts/eval_hf.py; the comparison plot lives in notebooks/04_eval_compare.ipynb.

License

Apache 2.0. The base model carries the Gemma terms of use; this adapter is bound by them too.