File size: 3,324 Bytes
695150a 1242644 9077358 695150a 9077358 ae89081 461a822 9077358 461a822 9077358 ae89081 9077358 1db100e 1242644 695150a 9077358 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | ---
license: other
license_name: see-each-subdirectory
tags: [tensor-decomposition, svd, low-rank, model-compression, knowledge-distillation]
---
# TR-3 study models: main compressed models and matched-budget baselines
Layout: **<method> / <stage> / <model>**. Method families are `tr3` (our 3-site tensor ring +
sparse residual), `tt3` (3-site tensor train ablation), `svd_llm`, `svd_llm_v2` and `asvd_llm`
(matched-budget baselines). Stages are `heal`
(KD-distilled, deployable) and `unheal` (raw post-compression, for measuring the healing delta).
| method | stage | model | params (B) | licence | KD result (probe) |
|---|---|---|---:|---|---|
| tr3 | heal | [`qwen2.5-7b`](tr3/heal/qwen2.5-7b) | 5.187 | qwen-research (non-commercial) | see dir |
| tt3 | heal | [`llama2-7b`](tt3/heal/llama2-7b) | 4.021 | llama2 | see dir |
| tt3 | heal | [`qwen2.5-3b`](tt3/heal/qwen2.5-3b) | 2.388 | qwen-research (non-commercial) | see dir |
| tt3 | unheal | [`llama2-7b`](tt3/unheal/llama2-7b) | 4.021 | llama2 | — |
| tt3 | unheal | [`qwen2.5-3b`](tt3/unheal/qwen2.5-3b) | 2.388 | qwen-research (non-commercial) | — |
| tr3 | heal | [`llama2-7b`](tr3/heal/llama2-7b) | 4.023 | llama2 | MMLU 43.21* |
| tr3 | heal | [`llama2-13b`](tr3/heal/llama2-13b) | 7.740 | llama2 | see dir |
| tr3 | heal | [`qwen2.5-3b`](tr3/heal/qwen2.5-3b) | 2.078 | qwen-research (non-commercial) | see dir |
| tr3 | unheal | [`llama2-13b`](tr3/unheal/llama2-13b) | 7.740 | llama2 | — |
| tr3 | unheal | [`llama2-7b`](tr3/unheal/llama2-7b) | (off-cluster archive) | llama2 | — |
| svd_llm | heal | [`llama2-7b`](svd_llm/heal/llama2-7b) | 4.022 | llama2 | MMLU 39.47 |
| svd_llm | heal | [`llama2-13b`](svd_llm/heal/llama2-13b) | 7.738 | llama2 | MMLU 46.54 |
| svd_llm | unheal | [`llama2-7b`](svd_llm/unheal/llama2-7b) | 4.022 | llama2 | — |
| svd_llm | unheal | [`llama2-13b`](svd_llm/unheal/llama2-13b) | 7.738 | llama2 | — |
| svd_llm_v2 ‡ | heal | [`llama2-7b`](svd_llm2/heal/llama2-7b) | 4.023 | llama2 | MMLU 40.22 |
| svd_llm_v2 ‡ | heal | [`llama2-13b`](svd_llm2/heal/llama2-13b) | 7.740 | llama2 | MMLU 44.96 |
| svd_llm_v2 ‡ | unheal | [`llama2-7b`](svd_llm2/unheal/llama2-7b) | 4.023 | llama2 | — |
| svd_llm_v2 ‡ | unheal | [`llama2-13b`](svd_llm2/unheal/llama2-13b) | 7.740 | llama2 | — |
| asvd_llm | heal | [`llama2-7b`](asvd_llm/heal/llama2-7b) | 4.023 | llama2 | MMLU 35.09 |
| asvd_llm | heal | [`llama2-13b`](asvd_llm/heal/llama2-13b) | 7.736 | llama2 | MMLU 42.46 |
| asvd_llm | unheal | [`llama2-7b`](asvd_llm/unheal/llama2-7b) | 4.023 | llama2 | — |
| asvd_llm | unheal | [`llama2-13b`](asvd_llm/unheal/llama2-13b) | 7.736 | llama2 | — |
\* full-benchmark lm-eval 0.4.12 result; SVD probe columns are limit-40 healing probes here —
each dir's README/provenance carries the precise protocol.
‡ Official SVD-LLM V2 code was never released; `svd_llm_v2` artifacts are our faithful
implementation of the published algorithms (per-layer ranks document the allocation).
## ⚠️ These are NOT standard `transformers` checkpoints
`tr3/**` stores **ring cores** (load with each dir's `tensor_formats.py`); `svd_llm*/**` stores
**low-rank factor pairs** (load with each dir's `lowrank_linear.py`). Weight files are
uniformly named `state_dict_fp16.pt`. `from_pretrained()` will not work on this repo.
|