Views
No views yet
A0 = plain finetune, evaluated uncompressed (the finetuning floor). A3 = in-loop
compression-aware finetune (the proposed method). The fair comparison is A3 vs A2a (A0's weights +
a fresh post-hoc SVD basis), at matched cache budget. Headline finding: A3 beats A2a by 2-8x PPL
across models and ratios. Keys are truncated pre-RoPE, values directly; rank = head_dim / ratio.| Subfolder | Model | Arm | Ratio | Notes |
|---|---|---|---|---|
out_llama/A3 | Llama-3.2-1B | A3 in-loop | 4x | STE gradient, full-FT (headline) |
out_llama_matrix/A3/r3 | Llama-3.2-1B | A3 in-loop | 3x | STE, full-FT; bases/ cover r3-r8 |
out_audits/A3_4x_exact | Llama-3.2-1B | A3 in-loop | 4x | EXACT projection gradient (STE-vs-exact ablation) |
out_llama_lc4096/A3 | Llama-3.2-1B | A3 in-loop | 4x | trained at seq-4096 (long-context fix) |
out_llama3_3b/{A0,A3} | Llama-3.2-3B | A0, A3 | 4x | LoRA adapters (merge onto the base) |
out_llama3_8b/{A0,A3} | Llama-3-8B | A0, A3 | 4x | LoRA adapters (merge onto the base) |
out_downstream/*.jsonl | -- | -- | -- | downstream MC-accuracy eval results |
bases/*.pt are the frozen SVD bases (base = from the base model, a0/ft = from the finetuned
weights). LoRA folders are adapters: load the matching base model and merge. Full per-checkpoint PPL
and downstream numbers are in the project's RESULTS.md.