loaiabdalslam/QSP22M-FRIDGEv3

QSP FRIDGE v3 — Qwen Statistical Subspace Projection (Kronecker Compressed)

Welcome to the QSP FRIDGE v3 repository. This project implements a pure Statistical Subspace Projection (QSP) engine that transfers the representational geometry of a large Qwen teacher model into an ultra-compact (~10M parameter) Qwen student with a 125k context window — using no gradients during projection. The student is then compressed with Nearest Kronecker Product (NKP) parameterization ("weights within weights") and optionally refined with lightweight Knowledge Distillation.

---

Mathematical Philosophy: Subspace Projection + Kronecker Compression

Standard parameter transfer fails when the teacher and student live in different dimensional spaces. QSP bypasses this by learning a shared subspace basis from the teacher's own embedding geometry and projecting every weight through it:

1. Learning the Shared Hidden Basis (SVD on the embedding manifold): $$E_{\text{teacher}} \approx U \cdot S \cdot V^T, \qquad R = V[:, :d_{\text{student}}]$$

2. Projecting Teacher Weights into Student Dimensions: For a 2D weight $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, the hidden axes are projected through the shared basis $R$, while non-hidden axes use fast bilinear interpolation: $$W_{\text{student}} = R^T , W_{\text{teacher}} , R$$

3. Kronecker Factorization (Weights Within Weights): Each projected matrix is stored as a rank-$k$ sum of Kronecker products, collapsing storage from $O(d_{\text{out}} d_{\text{in}})$ to $O(k(o_1 i_1 + o_2 i_2))$: $$W \approx \sum_{r=1}^{k} A_r \otimes B_r$$

4. Optional Knowledge Distillation: The Kronecker factors are fine-tuned to mimic the teacher's soft distribution at temperature $T$, guarded against vocab-dimension mismatch: $$\mathcal{L} = \alpha \cdot T^2 \cdot \text{KL}(p_{\text{student}}^{T} ,|, p_{\text{teacher}}^{T}) + (1-\alpha) \cdot \text{CE}(p_{\text{student}}, y)$$

---

Technical Performance Benchmark & Comprehensive Comparison

Below is the comparative report profiling every stage of the pipeline against the original Teacher (Qwen/Qwen2.5-0.5B) and the random baseline (Kronecker rank = 4):

1. Global Metrics Table

Model Under Test	Stored Params	Effective Params	Size (MB)	Eval Loss ↓	Perplexity (PPL) ↓	Latency (s) ↓	Speed (Tokens/Sec) ↑
Random-init Student	22.99M	22.99M	87.7	11.9956	162034.7859	N/As	N/A
Projected Student (QSP)	22.99M	22.99M	87.7	11.0649	63887.6343	0.6994s	85.79
Kronecker Student (QSP+Kron)	270.8K	22.99M	1.03	12.7544	346085.7305	1.8208s	32.95
Distilled Student (Kron+KD)	270.8K	22.99M	1.03	2.2301	9.3007	1.7455s	34.37
Teacher (Reference)	494.03M	494.03M	1884.59	2.405	11.0788	1.8175s	33.01

Kronecker Compression Ratio: ×85 (effective capacity retained at a fraction of stored parameters).

---

2. Output Generational Preview

Test Prompt:

"Machine learning is transforming the world by"

Projected Student (QSP) Generation:

Machine learning is transforming the world byitions restropros致ertoolsestrertositiontogrouptogroupitablyellarertoolsarrerrr aboutiotervertz AUTHORS itertools cm restert im (2.11-3.,/ (, - im arrrrrr val.ar implaitarm值ervarm.confitorerviewatioertoolsilloilloViewControllerAnimated

Distilled Student (Kron+KD) Generation:

Machine learning is transforming the world by certain., and energy (, the data,.., the sequence.. . the newspaper before work together to read a long library that work as the next word value back to set., the sequence. for every set. . the dataset in Python

, , and parameters on

---

🚀 Production Deployment & Evaluation Instructions

The final student is materialized into a standard Qwen2 checkpoint (model.safetensors + config.json), so it loads with the normal Hugging Face API and evaluates directly with lighteval / lm-eval-harness — no custom classes required.

Load with transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("./qsp_fridge_v3_production")
model = AutoModelForCausalLM.from_pretrained("./qsp_fridge_v3_production")

Evaluate with lighteval

lighteval accelerate \
  "pretrained=./qsp_fridge_v3_production" \
  "leaderboard|arc:challenge|0|0,leaderboard|hellaswag|0|0,leaderboard|winogrande|0|0"

Evaluate with lm-eval-harness

lm_eval --model hf \
  --model_args pretrained=./qsp_fridge_v3_production \
  --tasks arc_challenge,hellaswag,winogrande,truthfulqa_mc2,gsm8k,mmlu \
  --device cuda:0