Qwen3.5-2B SFT for LM Playschool Challenge (iter 2)

Headline results

Training methodology

Data usage

• Source: colab-potsdam/playpen-data, interactions split, train portion (34,909 episodes total).
• Filter: kept only episodes with meta.outcome == "success" → 20,202 episodes.
• Eval split: none. Iter 1 used a 20% holdout for in-training eval; iter 2 disabled in-training eval (it OOMed on A10G; eval-mode batches don't use gradient checkpointing). All 20,202 episodes used for training.
• Tokenization: Qwen's chat template applied. Sequences truncated to max_length=1024, which covers ~82% of the dataset uncut (verified empirically — see scripts/dataset_length_stats.py; iter 1 used 300, which truncated 64% of episodes — see iter-1 writeup).
• No external data, no test contamination. Only the public playpen-data train split was used.

Hyperparameters

Compute budget

Reproducibility

• scripts/setup.sh — clones playpen + clembench, builds venv, installs deps, applies our config patches. Idempotent.
• scripts/train_sft.py — the full trainer (CONFIG block at top exposes every hyperparameter).
• scripts/merge_and_push.py — LoRA merge + HF Hub upload.
• scripts/dataset_length_stats.py — episode-length analysis used to pick max_length.
• scripts/post_train_pipeline.sh — end-to-end orchestrator (waits on training pid → merge → push → eval).
• RUNS/iter1.md — iter 1 writeup with the failure analysis that motivated iter 2's hyperparameters.
• results/iter2.md — full per-game / per-benchmark breakdown of this model.

Design decisions

1. modules_to_save=[] instead of upstream's ["lm_head", "embed_token"]. This was the load-bearing fix between iter 1 (regression to clemscore 3.86) and iter 2 (gain to 46.66). Reasons:
   • Typo bug: the upstream string is embed_token (singular). Qwen's actual module is embed_tokens (plural), so peft silently ignored that entry — only lm_head was made trainable.
   • Tied-weight breakage: Qwen3.5 ships with tie_word_embeddings=True (the input embedding is the output head). Adding lm_head to modules_to_save causes peft to set tie_word_embeddings=False and create a separate trainable copy. After thousands of steps the head drifts away from the still-tied embedding, breaking an architectural invariant the model was pretrained against.
   • With modules_to_save=[], training is pure low-rank LoRA — no full-rank parameters touched, tied weights preserved.
2. max_length=1024 instead of upstream's 300. Empirical: median episode length in the success-filtered dataset is 462 tokens, p75 is 814, p90 is 1373. At 300, ~64% of training sequences had their assistant turn truncated; with TRL's effective full-sequence loss, that's ~64% of training compute spent on prefixes that never reached the target. Bumping to 1024 covers ~82% of episodes uncut. 2048 would cover ~95% but costs ~4× attention memory; we left the headroom for iter 3.
3. eval_strategy="no" (no in-training eval). TRL's eval loop disables gradient checkpointing for eval batches. On 4× A10G this OOMed at the first epoch boundary (iter 2's first attempt crashed at step 2021/6063 trying to allocate +7.58 GB on top of 15.85 GB already in use). We measure model quality via the full clembench eval after training, so eval_loss during training adds nothing.
4. save_strategy="steps", save_steps=1000, save_total_limit=2. Crash-recovery insurance. With save_strategy="epoch" the first checkpoint wouldn't appear until ~1.5 h in; on shared infrastructure that's a long uninsured stretch.
5. DDP via accelerate with a small monkeypatch in train_sft.py: clemcore's huggingface backend hardcodes device_map="auto", which pipeline-shards the model across all visible GPUs and runs only one GPU at a time. Under accelerate (LOCAL_RANK set), we override AutoModelForCausalLM.from_pretrained to use device_map={"": LOCAL_RANK} so each rank holds a full model copy on its own GPU. Single-process pipeline-parallel iter 1: 4 h. 4-GPU DDP iter 2: 2 h 36 m.
6. Generation-config fix on the merged model. Qwen3.5-2B's text_config.eos_token_id is 248044 (<|endoftext|>), but its chat template terminates with <|im_end|> (248046). After SFT, the model emitted <|im_end|> correctly but model.generate() didn't stop there (wrong eos_token_id), causing the model to "continue" past its turn and hallucinate fake user/assistant exchanges. Fixed by setting eos_token_id: [248046, 248044] and pad_token_id: 248044 in generation_config.json of the uploaded model. This is a property of the model, not an eval-pipeline trick.

Limitations and known regressions

• referencegame quality 100 → 8.33: largest per-game regression. SFT learned a verbose explanation style from playpen-data that hurts on the single-turn exact-match protocol of referencegame.
• ifeval 65.22 → 36.96, eqbench 65.01 → 46.77: SFT softened the model's instruction-following discipline. The base model's strict format adherence on these benches partially eroded.
• wordle unchanged at 33.33 % played, 0 quality: the model now follows wordle's protocol but doesn't have the world-model to actually win wordle.

License