loracle-pretrain-v7-sweep-A-step625
Step-625 (10% of epoch 1) checkpoint from v7 sweep A training run.
Training config
- Base: Qwen3-14B (frozen)
- Interpreter LoRA: rank=256, lora_alpha=32, rslora=True (effective scaling alpha/sqrt(rank)=2.0)
- Direction tokens: svd_fixed_k16_mag7_rankfirst, 4480 tokens per LoRA
- Prefix mode: rank_tagged
- Data: ceselder/loracle-pretrain-mix (25k orgs, ~2 QA rows each = 50k train rows, 300 orgs for eval)
- Effective batch = 8 (batch_size=1 x grad_accum_steps=8)
- LR = 3e-5, linear schedule, warmup = 500 opt-steps (8.9% of training)
- Epochs = 1 (target 6,250 opt-steps total; step-625 = 10% mark)
Eval numbers at step 625
Judge: Sonnet 4.6 via OpenRouter.
| Set | organisms | any-match | rollout-mean |
|---|
| heldout_ia | 20 | 30.0% | 8.3% |
| trigger_recovery_heldout_ia | 20 | 20.0% | 10.0% |
| auditbench | 56 | 0.0% | 0.0% |
| ood_models_v3 | 27 | 33.3% | 11.4% |
| val/mean_all_evals | - | 20.8% | - |
Caveat: auditbench 0% was under an overly strict custom rubric that has
since been reverted to the canonical IA-paper rubric. Rejudging the saved
rollouts under the standard rubric should yield a much higher AB number
(v6_A at a comparable stage was 60%+ under the paper rubric).
Wandb
Layout
- interpreter/ PEFT LoRA adapter (load with PeftModel.from_pretrained)
- encoder.pt AO encoder state_dict
- ao.pt AO norm-match hook params
- tokenizer/ Qwen3-14B tokenizer
- loracle_config.yaml Training config snapshot
