You can deploy this model and start earning money today!

roozbehn99/
palm-qwen3b-sky-brev-seed5

Name: palm-qwen3b-sky-brev-seed5
Availability: InStock

Qwen/Qwen2.5-3B-Instruct

finetune

apache-2.0

Views

No views yet

PALM portfolio — Qwen2.5-3B, helpfulness (Skywork RM) × brevity, training seed 5

13 policies fine-tuned with GRPO from Qwen/Qwen2.5-3B-Instruct, one per weight vector on the 2-objective simplex (helpfulness reward model + brevity verifiable reward). Weight vectors are the 13 points produced by Algorithm 1 of the PALM paper (ε=0.4, δ=0.2). This repo is one training seed (seed=5) of a multi-seed study; sibling repos hold the other seeds with identical configuration.

Every model is a full-parameter fine-tune stored in HF transformers format (bf16), each in its own subfolder idx0/ … idx12/.

Load a model

python

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "roozbehn99/palm-qwen3b-sky-brev-seed5"
tok = AutoTokenizer.from_pretrained(repo, subfolder="idx6")   # (0.5, 0.5) model
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="idx6", torch_dtype="bfloat16")

Models, weights, and post-hoc evaluation

R1/R2 are min-max-normalized rewards on the RLVR-GSM test set (1,319 prompts, sampled decoding T=0.7, 256 max new tokens); KL is the token-summed KL to the reference model. The scalarized objective is J_w(π) = w·R(π) − β·KL(π) with β = 0.05. Machine-readable copies: weights.csv, eval_summary.csv.

subfolder	w_help	w_brev	R1 (help)	R2 (brev)	KL
`idx0`	1.000	0.000	0.800	0.184	0.63
`idx1`	0.833	0.167	0.831	0.224	0.86
`idx2`	0.781	0.219	0.837	0.248	1.05
`idx3`	0.718	0.282	0.839	0.282	1.35
`idx4`	0.646	0.354	0.839	0.310	1.60
`idx5`	0.566	0.434	0.810	0.326	2.05
`idx6`	0.500	0.500	0.787	0.352	2.57
`idx7`	0.000	1.000	0.722	0.435	5.45
`idx8`	0.167	0.833	0.720	0.416	4.97
`idx9`	0.219	0.781	0.746	0.387	4.21
`idx10`	0.282	0.718	0.754	0.385	3.54
`idx11`	0.354	0.646	0.757	0.388	3.57
`idx12`	0.434	0.566	0.789	0.372	3.21

Training configuration (identical for all models)

policy: Qwen/Qwen2.5-3B-Instruct
reference: Qwen/Qwen2.5-3B-Instruct
rm: Skywork/Skywork-Reward-Llama-3.1-8B (helpfulness; R1)
vr: brevity verifiable reward (open_instruct/rlvr_objectives.py::brevity; R2)
normalization: both rewards min-max calibrated to [0,1] at training time (rm_calibrations/Skywork__Skywork-Reward-Llama-3.1-8B/stats.json, rm_calibrations/vr__brevity__Qwen__Qwen2.5-3B-Instruct/stats.json)
algorithm: GRPO (open_instruct/weighted_grpo.py), scalar reward r = w1R1 + w2R2
beta: 0.01 (KL coefficient vs. reference policy)
data: allenai/RLVR-GSM + allenai/RLVR-MATH (train); allenai/RLVR-GSM test (eval)
episodes: 40,000 (208 optimizer steps)
lr: 5e-7
batch: 4 x grad_acc 4, 4 samples/prompt
response_len: 256 tokens
chat_template: tokenizer native (Qwen)
weights: Algorithm 1 of the paper (epsilon=0.4, delta=0.2) -> 13 weight vectors on the 2-simplex
hardware: 4x NVIDIA A100-80GB (OSC Ascend, 3 training + 1 vLLM) or 4x H100 (OSC Cardinal)
precision: bf16

Provenance

Code: LMPortfolio (open_instruct/weighted_grpo.py, open_instruct/make_weights.py) — a fork of AI2's open-instruct.
Portfolio directory on the training cluster: portfolio_runs/qwen3b_sky_brev_iclr_s5.
Training seed: --seed 5 (Python / NumPy / Torch RNGs; per-process offset by rank).
Reward calibration statistics are shipped with the code repository (rm_calibrations/).

Intended use

Research artifact for multi-objective alignment / portfolio pruning experiments. Not safety-tuned beyond the base model; not intended for deployment.