| --- |
| license: other |
| license_name: see-each-subdirectory |
| tags: [tensor-decomposition, svd, low-rank, model-compression, knowledge-distillation] |
| --- |
| |
| # TR-3 study models: main compressed models and matched-budget baselines |
|
|
| Layout: **<method> / <stage> / <model>**. Method families are `tr3` (our 3-site tensor ring + |
| sparse residual), `tt3` (3-site tensor train ablation), `svd_llm`, `svd_llm_v2` and `asvd_llm` |
| (matched-budget baselines). Stages are `heal` |
| (KD-distilled, deployable) and `unheal` (raw post-compression, for measuring the healing delta). |
|
|
| | method | stage | model | params (B) | licence | KD result (probe) | |
| |---|---|---|---:|---|---| |
| | tr3 | heal | [`qwen2.5-7b`](tr3/heal/qwen2.5-7b) | 5.187 | qwen-research (non-commercial) | see dir | |
| | tt3 | heal | [`llama2-7b`](tt3/heal/llama2-7b) | 4.021 | llama2 | see dir | |
| | tt3 | heal | [`qwen2.5-3b`](tt3/heal/qwen2.5-3b) | 2.388 | qwen-research (non-commercial) | see dir | |
| | tt3 | unheal | [`llama2-7b`](tt3/unheal/llama2-7b) | 4.021 | llama2 | — | |
| | tt3 | unheal | [`qwen2.5-3b`](tt3/unheal/qwen2.5-3b) | 2.388 | qwen-research (non-commercial) | — | |
| | tr3 | heal | [`llama2-7b`](tr3/heal/llama2-7b) | 4.023 | llama2 | MMLU 43.21* | |
| | tr3 | heal | [`llama2-13b`](tr3/heal/llama2-13b) | 7.740 | llama2 | see dir | |
| | tr3 | heal | [`qwen2.5-3b`](tr3/heal/qwen2.5-3b) | 2.078 | qwen-research (non-commercial) | see dir | |
| | tr3 | unheal | [`llama2-13b`](tr3/unheal/llama2-13b) | 7.740 | llama2 | — | |
| | tr3 | unheal | [`llama2-7b`](tr3/unheal/llama2-7b) | (off-cluster archive) | llama2 | — | |
| | svd_llm | heal | [`llama2-7b`](svd_llm/heal/llama2-7b) | 4.022 | llama2 | MMLU 39.47 | |
| | svd_llm | heal | [`llama2-13b`](svd_llm/heal/llama2-13b) | 7.738 | llama2 | MMLU 46.54 | |
| | svd_llm | unheal | [`llama2-7b`](svd_llm/unheal/llama2-7b) | 4.022 | llama2 | — | |
| | svd_llm | unheal | [`llama2-13b`](svd_llm/unheal/llama2-13b) | 7.738 | llama2 | — | |
| | svd_llm_v2 ‡ | heal | [`llama2-7b`](svd_llm2/heal/llama2-7b) | 4.023 | llama2 | MMLU 40.22 | |
| | svd_llm_v2 ‡ | heal | [`llama2-13b`](svd_llm2/heal/llama2-13b) | 7.740 | llama2 | MMLU 44.96 | |
| | svd_llm_v2 ‡ | unheal | [`llama2-7b`](svd_llm2/unheal/llama2-7b) | 4.023 | llama2 | — | |
| | svd_llm_v2 ‡ | unheal | [`llama2-13b`](svd_llm2/unheal/llama2-13b) | 7.740 | llama2 | — | |
| | asvd_llm | heal | [`llama2-7b`](asvd_llm/heal/llama2-7b) | 4.023 | llama2 | MMLU 35.09 | |
| | asvd_llm | heal | [`llama2-13b`](asvd_llm/heal/llama2-13b) | 7.736 | llama2 | MMLU 42.46 | |
| | asvd_llm | unheal | [`llama2-7b`](asvd_llm/unheal/llama2-7b) | 4.023 | llama2 | — | |
| | asvd_llm | unheal | [`llama2-13b`](asvd_llm/unheal/llama2-13b) | 7.736 | llama2 | — | |
|
|
| \* full-benchmark lm-eval 0.4.12 result; SVD probe columns are limit-40 healing probes here — |
| each dir's README/provenance carries the precise protocol. |
|
|
| ‡ Official SVD-LLM V2 code was never released; `svd_llm_v2` artifacts are our faithful |
| implementation of the published algorithms (per-layer ranks document the allocation). |
|
|
| ## ⚠️ These are NOT standard `transformers` checkpoints |
|
|
| `tr3/**` stores **ring cores** (load with each dir's `tensor_formats.py`); `svd_llm*/**` stores |
| **low-rank factor pairs** (load with each dir's `lowrank_linear.py`). Weight files are |
| uniformly named `state_dict_fp16.pt`. `from_pretrained()` will not work on this repo. |
|
|
