loracle-pretrain-v7-sweep-A-oneq-step1248
Step-1248 (40% of epoch 1) checkpoint from v7 sweep A "oneq" variant
(1 randomly-selected QA per organism, half the data of v7_A).
Training config
- Base: Qwen3-14B (frozen)
- Interpreter LoRA: rank=256, lora_alpha=32, rslora=True (effective scaling alpha/sqrt(rank)=2.0)
- Direction tokens: svd_fixed_k16_mag7_rankfirst, 4480 tokens per LoRA
- Prefix mode: rank_tagged
- Data: ceselder/loracle-pretrain-mix subsampled to 1 random QA row per
organism (seed=42 deterministic shuffle+drop_duplicates) → 25k train rows
- Eval: 300 holdout orgs (1 row each)
- Effective batch = 8 (batch_size=1 x grad_accum_steps=8)
- LR = 3e-5, linear schedule, warmup = 500 opt-steps
- Epochs = 1, total 3125 opt-steps; this checkpoint at step 1248 = 40% mark
Eval numbers at step 1248
Judge: Sonnet 4.6 via OpenRouter, canonical IA-paper rubric.
| Set | organisms | any-match |
|---|
| heldout_ia | 20 | 40.0% |
| trigger_recovery_heldout_ia | 20 | 20.0% |
| auditbench | 56 | 26.8% |
| ood_models_v3 | 27 | 33.3% |
| val/mean_all_evals | - | 30.0% |
Trajectory of the oneq run so far
| step (% epoch) | mean |
|---|
| 312 (10%) | 16.3% |
| 624 (20%) | 27.9% |
| 936 (30%) | 26.1% |
| 1248 (40%) | 30.0% |
| 1560 (50%) | 32.7% (later peak) |
Wandb
Layout
- interpreter/ PEFT LoRA adapter
- encoder.pt AO encoder state_dict
- ao.pt AO norm-match hook params
- tokenizer/ Qwen3-14B tokenizer
- loracle_config.yaml Training config snapshot
