Loracle — pretrain v4 sweep A, full epoch (step 2598)

• Base model: Qwen/Qwen3-14B
• Interpreter adapter: rank-256 PEFT LoRA with rslora=True, α=32
• Direction-token input: 4480-token prefix in (rank, layer, mag7) rank-first layout, produced by extract_svd_fixed_tokens from any Qwen3-14B LoRA / full-FT / GGUF (via extract_hf/download_and_extract_hf_models.py).
• Training corpus: 2500 synthetic pretrain organisms (fineweb + redpajama2 + misc) from ceselder/loracle-pretrain-qa-v4.1-25k, one QA pair per organism × 2 = ~5000 training examples.
• Hook: layer-1 AO norm-match injection (h' = h + ‖h‖ · v̂).
• Prefix mode: rank_tagged (16 <SVD N:> markers × 280 placeholder slots per rank).

wandb run

Eval numbers (at step 2598)

• val/loss: 1.907
• val/mean_all_evals: 32.3%
• auditbench rollout_mean: 19.9%
• heldout_ia rollout_mean: 21.1%
• ood_models_v3 rollout_mean: 14.4%
• trigger_recovery: 0% (keyword-match judge was too strict; use semantic rubric in newer eval yamls)

Repo contents

• interpreter/ — PEFT adapter (load with PeftModel.from_pretrained(base, "interpreter"))
• tokenizer/ — HF tokenizer (inherits Qwen3-14B's, no vocab extension)
• encoder.pt — AOEncoder state-dict (no learnable params; kept for API symmetry)
• ao.pt — training metadata (step, val_loss, val_mean_all_evals)
• loracle_config.yaml — full training config snapshot

Loading