Rhett1 commited on
Commit
b2209bd
·
verified ·
1 Parent(s): 1dc269d

Upload svd_llm2/unheal/llama2-7b-base/README.md with huggingface_hub

Browse files
svd_llm2/unheal/llama2-7b-base/README.md ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: llama2
3
+ base_model: NousResearch/Llama-2-7b-hf
4
+ tags: [svd, low-rank, model-compression, llama2]
5
+ ---
6
+
7
+ # SVD-LLM V2 (NAACL 2025, official code unreleased; our implementation of Alg. 1+2) @ 4.02B, UNHEALED (LLaMA-2-7B base)
8
+
9
+ - **UNHEALED checkpoint**: raw post-compression weights, NO distillation. The KD-healed
10
+ counterpart is at `../heal/llama2-7b-base/` (same `ranks.json`; weights differ).
11
+ - Factored params: **4,023,134,464** (budget 4.0231B, matched to the TR-3 llama2-7b row;
12
+ SVD-LLM V2 overall keep-ratio 0.5680 across the factored projections).
13
+ - probe MMLU (limit 40/subtask, n=2280, seed 1234, fp16 — measured as the step-0 eval of
14
+ the sibling heal run): **24.34%**, i.e. essentially chance level; FP16 dense 7B base
15
+ teacher on the same probe: 47.41%. Compression alone, without healing, destroys base-model
16
+ instruction-independent capability at this budget.
17
+
18
+ Load with `lowrank_linear.py` in this dir (identical recipe to the sibling dirs).
19
+ Compression-provenance details (source revision, sha256 of every artifact, allocation
20
+ files) are in `provenance.json`.