You can deploy this model and start earning money today!

zhc12/
specturn-llama2-7b

Name: specturn-llama2-7b
Availability: InStock

meta-llama/Llama-2-7b-hf

finetune

llama2

Views

No views yet

SpecTurn — Compression-Aware Dense Training for LLaMA-2-7B

SVD-compressed LLaMA-2-7B models from the SpecTurn pipeline (compression ratio 0.2).

Results

Model	WikiText-2 PPL	Method
LLaMA-2-7B (dense)	5.47	—
SVD-LLM(W)	8.38	Whitened SVD truncation
DynRank	7.63	Factor training + dynamic rank
SpecTurn Phase 1	6.51	Shadow-guided dense training
SpecTurn Phase 1+2	6.18	+ Factor fine-tuning with KL distillation

Models

`phase2-best-r02/`

Best model (PPL=6.18). Merged HF format, ready for inference.

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "zhc12/specturn-llama2-7b",
    subfolder="phase2-best-r02",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

`phase1-dense-teacher/`

Phase 1 dense teacher model (PPL=5.53 dense, 6.51 after SVD truncation). Used as KL distillation target in Phase 2.

`phase2-factors-r02/`

Phase 2 factor checkpoint: factors.pt (224 modules × (U, s, V)), allocator.pt, rank_allocation.json. For research use.

Citation

bibtex

@article{specturn2026,
  title={Training Dense Weights for Low-Rank Compression: Shadow-Guided Spectral Restructuring},
  author={Zhang, Huicheng},
  year={2026},
}