zhc12/specturn-llama2-7b

SpecTurn — Compression-Aware Dense Training for LLaMA-2-7B

SVD-compressed LLaMA-2-7B models from the SpecTurn pipeline (compression ratio 0.2).

Results

Model	WikiText-2 PPL	Method
LLaMA-2-7B (dense)	5.47	—
SVD-LLM(W)	8.38	Whitened SVD truncation
DynRank	7.63	Factor training + dynamic rank
SpecTurn Phase 1	6.51	Shadow-guided dense training
SpecTurn Phase 1+2	6.18	+ Factor fine-tuning with KL distillation

Models

phase2-best-r02/

Best model (PPL=6.18). Merged HF format, ready for inference.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "zhc12/specturn-llama2-7b",
    subfolder="phase2-best-r02",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

phase1-dense-teacher/

Phase 1 dense teacher model (PPL=5.53 dense, 6.51 after SVD truncation). Used as KL distillation target in Phase 2.

phase2-factors-r02/

Phase 2 factor checkpoint: factors.pt (224 modules × (U, s, V)), allocator.pt, rank_allocation.json. For research use.

Citation

@article{specturn2026,
  title={Training Dense Weights for Low-Rank Compression: Shadow-Guided Spectral Restructuring},
  author={Zhang, Huicheng},
  year={2026},
}