You can deploy this model and start earning money today!

roozbehn99/
palm-qwen3b-sky-brev-seed2

Name: palm-qwen3b-sky-brev-seed2
Availability: InStock

Qwen/Qwen2.5-3B-Instruct

finetune

apache-2.0

Views

No views yet

PALM portfolio — Qwen2.5-3B, helpfulness (Skywork RM) × brevity, training seed 2

13 policies fine-tuned with GRPO from Qwen/Qwen2.5-3B-Instruct, one per weight vector on the 2-objective simplex (helpfulness reward model + brevity verifiable reward). Weight vectors are the 13 points produced by Algorithm 1 of the PALM paper (ε=0.4, δ=0.2). This repo is one training seed (seed=2) of a multi-seed study; sibling repos hold the other seeds with identical configuration.

Every model is a full-parameter fine-tune stored in HF transformers format (bf16), each in its own subfolder idx0/ … idx12/.

Load a model

python

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "roozbehn99/palm-qwen3b-sky-brev-seed2"
tok = AutoTokenizer.from_pretrained(repo, subfolder="idx6")   # (0.5, 0.5) model
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="idx6", torch_dtype="bfloat16")

Models, weights, and post-hoc evaluation

R1/R2 are min-max-normalized rewards on the RLVR-GSM test set (1,319 prompts, sampled decoding T=0.7, 256 max new tokens); KL is the token-summed KL to the reference model. The scalarized objective is J_w(π) = w·R(π) − β·KL(π) with β = 0.05. Machine-readable copies: weights.csv, eval_summary.csv.

subfolder	w_help	w_brev	R1 (help)	R2 (brev)	KL
`idx0`	1.000	0.000	0.797	0.179	0.59
`idx1`	0.833	0.167	0.831	0.226	0.85
`idx2`	0.781	0.219	0.828	0.239	0.81
`idx3`	0.718	0.282	0.827	0.278	1.23
`idx4`	0.646	0.354	0.823	0.306	1.51
`idx5`	0.566	0.434	0.805	0.344	2.19
`idx6`	0.500	0.500	0.792	0.358	2.77
`idx7`	0.000	1.000	0.689	0.456	6.13
`idx8`	0.167	0.833	0.728	0.406	4.53
`idx9`	0.219	0.781	0.738	0.393	3.94
`idx10`	0.282	0.718	0.742	0.409	4.48
`idx11`	0.354	0.646	0.768	0.383	3.56
`idx12`	0.434	0.566	0.778	0.368	3.13

Training configuration (identical for all models)

policy: Qwen/Qwen2.5-3B-Instruct
reference: Qwen/Qwen2.5-3B-Instruct
rm: Skywork/Skywork-Reward-Llama-3.1-8B (helpfulness; R1)
vr: brevity verifiable reward (open_instruct/rlvr_objectives.py::brevity; R2)
normalization: both rewards min-max calibrated to [0,1] at training time (rm_calibrations/Skywork__Skywork-Reward-Llama-3.1-8B/stats.json, rm_calibrations/vr__brevity__Qwen__Qwen2.5-3B-Instruct/stats.json)
algorithm: GRPO (open_instruct/weighted_grpo.py), scalar reward r = w1R1 + w2R2
beta: 0.01 (KL coefficient vs. reference policy)
data: allenai/RLVR-GSM + allenai/RLVR-MATH (train); allenai/RLVR-GSM test (eval)
episodes: 40,000 (208 optimizer steps)
lr: 5e-7
batch: 4 x grad_acc 4, 4 samples/prompt
response_len: 256 tokens
chat_template: tokenizer native (Qwen)
weights: Algorithm 1 of the paper (epsilon=0.4, delta=0.2) -> 13 weight vectors on the 2-simplex
hardware: 4x NVIDIA A100-80GB (OSC Ascend, 3 training + 1 vLLM) or 4x H100 (OSC Cardinal)
precision: bf16

Provenance

Code: LMPortfolio (open_instruct/weighted_grpo.py, open_instruct/make_weights.py) — a fork of AI2's open-instruct.
Portfolio directory on the training cluster: portfolio_runs/qwen3b_sky_brev_iclr_s2.
Training seed: --seed 2 (Python / NumPy / Torch RNGs; per-process offset by rank).
Reward calibration statistics are shipped with the code repository (rm_calibrations/).

Intended use

Research artifact for multi-objective alignment / portfolio pruning experiments. Not safety-tuned beyond the base model; not intended for deployment.