license: other
license_name: see-each-subdirectory
tags:
- tensor-decomposition
- svd
- low-rank
- model-compression
- knowledge-distillation
TR-3 study models: main compressed models and matched-budget baselines
Layout: / / . Method families are tr3 (our 3-site tensor ring +
sparse residual), tt3 (3-site tensor train ablation), svd_llm, svd_llm_v2 and asvd_llm
(matched-budget baselines). Stages are heal
(KD-distilled, deployable) and unheal (raw post-compression, for measuring the healing delta).
| method | stage | model | params (B) | licence | KD result (probe) |
|---|---|---|---|---|---|
| tr3 | heal | qwen2.5-7b |
5.187 | qwen-research (non-commercial) | see dir |
| tt3 | heal | llama2-7b |
4.021 | llama2 | see dir |
| tt3 | heal | qwen2.5-3b |
2.388 | qwen-research (non-commercial) | see dir |
| tt3 | unheal | llama2-7b |
4.021 | llama2 | — |
| tt3 | unheal | qwen2.5-3b |
2.388 | qwen-research (non-commercial) | — |
| tr3 | heal | llama2-7b |
4.023 | llama2 | MMLU 43.21* |
| tr3 | heal | llama2-13b |
7.740 | llama2 | see dir |
| tr3 | heal | qwen2.5-3b |
2.078 | qwen-research (non-commercial) | see dir |
| tr3 | unheal | llama2-13b |
7.740 | llama2 | — |
| tr3 | unheal | llama2-7b |
(off-cluster archive) | llama2 | — |
| svd_llm | heal | llama2-7b |
4.022 | llama2 | MMLU 39.47 |
| svd_llm | heal | llama2-13b |
7.738 | llama2 | MMLU 46.54 |
| svd_llm | unheal | llama2-7b |
4.022 | llama2 | — |
| svd_llm | unheal | llama2-13b |
7.738 | llama2 | — |
| svd_llm_v2 ‡ | heal | llama2-7b |
4.023 | llama2 | MMLU 40.22 |
| svd_llm_v2 ‡ | heal | llama2-13b |
7.740 | llama2 | MMLU 44.96 |
| svd_llm_v2 ‡ | unheal | llama2-7b |
4.023 | llama2 | — |
| svd_llm_v2 ‡ | unheal | llama2-13b |
7.740 | llama2 | — |
| asvd_llm | heal | llama2-7b |
4.023 | llama2 | MMLU 35.09 |
| asvd_llm | heal | llama2-13b |
7.736 | llama2 | MMLU 42.46 |
| asvd_llm | unheal | llama2-7b |
4.023 | llama2 | — |
| asvd_llm | unheal | llama2-13b |
7.736 | llama2 | — |
* full-benchmark lm-eval 0.4.12 result; SVD probe columns are limit-40 healing probes here — each dir's README/provenance carries the precise protocol.
‡ Official SVD-LLM V2 code was never released; svd_llm_v2 artifacts are our faithful
implementation of the published algorithms (per-layer ranks document the allocation).
⚠️ These are NOT standard transformers checkpoints
tr3/** stores ring cores (load with each dir's tensor_formats.py); svd_llm*/** stores
low-rank factor pairs (load with each dir's lowrank_linear.py). Weight files are
uniformly named state_dict_fp16.pt. from_pretrained() will not work on this repo.