Update README.md

README.md

@@ -13,7 +13,6 @@ base_model:
 - MiniMaxAI/MiniMax-M2
 ---
 
-
 ![Screenshot](https://huggingface.co/VibeStudio/MiniMax-M2-THRIFT/resolve/main/vibe_processed_by_imagy.png)
 
 # THRIFT — Targeted Reduction for Inference and Fine-Tuning
@@ -22,27 +21,27 @@ A performance-optimized variant of the base model that delivers faster responses
 
 ## TLDR
 
-We, over-caffinated researchers at VibeStud.io wanted to create a 50% pruned version of the SOTA MiniMax M2 that is best suited for local/air-gapped coding. This version we achieved \~25%. A 50% pruned version is under development while a not so sucky team of ours is working on a  50% pruned version of Kimi K2 Thinking.We’re writing the paper and expanding the evaluation set to substantiate the results. Check back later, cheers\!
+We, over-caffinated researchers at VibeStud.io wanted to create a 50% pruned version of the SOTA MiniMax M2 that is best suited for local/air-gapped coding. This version we achieved ~25%. A 50% pruned version is under development while a not so sucky team of ours is working on a  50% pruned version of Kimi K2 Thinking.We’re writing the paper and expanding the evaluation set to substantiate the results. Check back later, cheers!
 
 ## Why it’s useful
 
-* **Lower latency:** Snappier responses for interactive apps and chatbots.  
-* **Smaller memory footprint:** Runs on cheaper GPUs or with fewer resources per replica.  
-* **Higher throughput:** Serve more concurrent users at the same cost.  
-* **Deployment-friendly:** Drop-in replacement for the base model in most inference stacks.  
+* **Lower latency:** Snappier responses for interactive apps and chatbots.
+* **Smaller memory footprint:** Runs on cheaper GPUs or with fewer resources per replica.
+* **Higher throughput:** Serve more concurrent users at the same cost.
+* **Deployment-friendly:** Drop-in replacement for the base model in most inference stacks.
 * **Adaptable:** Supports light fine-tuning to match your domain and style guidelines.
 
 ## Intended use
 
-* General chat and coding assistance  
-* Enterprise assistants with strict latency/VRAM budgets  
-* Batch or realtime serving in cloud and on-prem environments  
+* General chat and coding assistance
+* Enterprise assistants with strict latency/VRAM budgets
+* Batch or realtime serving in cloud and on-prem environments
 * Edge or cost-sensitive deployments where efficiency matters
 
 ## When to use it
 
-* You’re constrained by GPU memory or need shorter response times  
-* You want to increase QPS without scaling infrastructure  
+* You’re constrained by GPU memory or need shorter response times
+* You want to increase QPS without scaling infrastructure
 * You need a model that is “good enough” for most tasks at a better cost profile
 
 ---
@@ -51,115 +50,108 @@ We, over-caffinated researchers at VibeStud.io wanted to create a 50% pruned ver
 
 **Models Under Evaluation**
 
-| Model | Type |
-| :---- | :---- |
-| ModelCloud/MiniMax-M2-BF16 | Base Model |
+| Model                        | Type                 |
+| :--------------------------- | :------------------- |
+| ModelCloud/MiniMax-M2-BF16   | Base Model           |
 | VibeStudio/MiniMax-M2-THRIFT | Compressed/Optimized |
 
-**Evaluation Date: November 7, 2025**
+**Evaluation Dates:** November 7–9, 2025
 
 ## 📊 Results Comparison
 
-### 1\) Multiple Choice Q\&A (lm-eval)
+### 1) Multiple Choice Q&A (lm-eval)
 
 **Overall MMLU Performance**
 
-| Model | MMLU Overall | Humanities | STEM | Social Sciences | Other |
-| :---- | ----: | ----: | ----: | ----: | ----: |
-| MiniMax-M2-BF16 | 83.16% | 77.45% | 80.91% | 90.02% | 87.29% |
-| MiniMax-M2-THRIFT | 77.72% | 70.14% | 77.61% | 86.84% | 80.27% |
-| **Δ (Difference)** | **\-5.44%** | **\-7.31%** | **\-3.30%** | **\-3.18%** | **\-7.02%** |
+| Model              | MMLU Overall | Humanities |   STEM | Social Sciences |  Other |
+| :----------------- | -----------: | ---------: | -----: | --------------: | -----: |
+| MiniMax-M2-BF16    |   **83.16%** |     77.45% | 80.91% |      **90.02%** | 87.29% |
+| MiniMax-M2-THRIFT  |   **77.72%** |     70.14% | 77.61% |          86.84% | 80.27% |
+| **Δ (Difference)** |   **-5.44%** |     -7.31% | -3.30% |          -3.18% | -7.02% |
 
 **Individual Task Performance**
 
-| Task | BF16 (Base) | THRIFT-BF16 | Difference |
-| :---- | ----: | ----: | ----: |
-| arc\_challenge | 73.21% | 61.01% | \-12.20% ⬇️ |
-| arc\_easy | 88.30% | 83.08% | \-5.22% ⬇️ |
-| boolq | 87.95% | 84.95% | \-3.00% ⬇️ |
-| hellaswag | 83.00% | 77.09% | \-5.91% ⬇️ |
-| mmlu | 83.16% | 77.72% | \-5.44% ⬇️ |
-| openbookqa | 48.60% | 43.00% | \-5.60% ⬇️ |
-| rte | 75.45% | 80.14% | **\+4.69% ⬆️** |
-| winogrande | 76.48% | 74.90% | \-1.58% ⬇️ |
+| Task                     | BF16 (Base) | THRIFT-BF16 |    Difference |
+| :----------------------- | ----------: | ----------: | ------------: |
+| arc_challenge (acc_norm) |      73.21% |      61.01% |    -12.20% ⬇️ |
+| arc_easy                 |      88.30% |      83.08% |     -5.22% ⬇️ |
+| boolq                    |      87.95% |      84.95% |     -3.00% ⬇️ |
+| hellaswag (acc_norm)     |      83.00% |      77.09% |     -5.91% ⬇️ |
+| mmlu                     |      83.16% |      77.72% |     -5.44% ⬇️ |
+| openbookqa (acc_norm)    |      48.60% |      43.00% |     -5.60% ⬇️ |
+| rte                      |      75.45% |  **80.14%** | **+4.69% ⬆️** |
+| winogrande               |      76.48% |      74.90% |     -1.58% ⬇️ |
 
-**Average Accuracy Drop: \-4.28%**
+**Average Accuracy Drop:** **-4.28%**
 
-### 2\) Code Generation (EvalPlus)
+### 2) Code Generation (EvalPlus)
 
-**MBPP Results**
+**MBPP Results (Python, 378 problems)**
 
-| Model | MBPP (base) | MBPP+ (extended) |
-| :---- | ----: | ----: |
-| MiniMax-M2-BF16 | 73.8% | 64.0% |
-| MiniMax-M2-THRIFT | 🔄 Coming Soon | 🔄 Coming Soon |
+| Model              | MBPP (base) | MBPP+ (extended) |   Average |
+| :----------------- | ----------: | ---------------: | --------: |
+| MiniMax-M2-BF16    |   **73.8%** |        **64.0%** |     68.9% |
+| MiniMax-M2-THRIFT  |   **70.1%** |        **60.1%** |     65.1% |
+| **Δ (Difference)** |   **-3.7%** |        **-3.9%** | **-3.8%** |
 
-**HumanEval Results**
+**HumanEval Results (164 problems)**
 
-| Model | HumanEval (base) | HumanEval+ (extended) |
-| :---- | ----: | ----: |
-| MiniMax-M2-BF16 | ✅ Complete | ✅ Complete |
-| MiniMax-M2-THRIFT | 🔄 Coming Soon | 🔄 Coming Soon |
+| Model              | HumanEval (base) | HumanEval+ (extended) |   Average |
+| :----------------- | ---------------: | --------------------: | --------: |
+| MiniMax-M2-BF16    |        **72.6%** |             **71.3%** |     72.0% |
+| MiniMax-M2-THRIFT  |        **65.2%** |             **63.4%** |     64.3% |
+| **Δ (Difference)** |        **-7.4%** |             **-7.9%** | **-7.7%** |
 
-### 3\) Math Benchmarks
+### 3) Math Benchmarks
 
 **GSM8K Results**
 
-| Model | Accuracy | Problems |
-| :---- | ----: | ----: |
-| MiniMax-M2-BF16 | 92.72% | 1,319 |
-| MiniMax-M2-THRIFT | 🔄 Coming Soon | 1,319 |
+| Model              |      Accuracy | Problems | Status               |
+| :----------------- | ------------: | -------: | :------------------- |
+| MiniMax-M2-BF16    |    **92.72%** |    1,319 | ✅ Complete           |
+| MiniMax-M2-THRIFT  |    **93.25%** |    1,319 | ✅ Complete           |
+| **Δ (Difference)** | **+0.53% ⬆️** |        - | **THRIFT Better!** ✨ |
 
 **MATH-500 Results**
 
-| Model | Overall | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 |
-| :---- | ----: | ----: | ----: | ----: | ----: | ----: |
-| MiniMax-M2-BF16 | 87.2% | 90.7% | 95.56% | 82.86% | 85.16% | 85.82% |
-| MiniMax-M2-THRIFT | 🔄 Coming Soon | 🔄 | 🔄 | 🔄 | 🔄 | 🔄 |
+| Model             |   Overall | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 | Status          |
+| :---------------- | --------: | ------: | ------: | ------: | ------: | ------: | :-------------- |
+| MiniMax-M2-BF16   | **87.2%** |   90.7% |  95.56% |  82.86% |  85.16% |  85.82% | ✅ Complete      |
+| MiniMax-M2-THRIFT |      🔄 — |      🔄 |      🔄 |      🔄 |      🔄 |      🔄 | **In Progress** |
 
-### 4\) LiveCodeBench (Live Coding Problems)
+### 4) LiveCodeBench (Live Coding Problems)
 
-| Model | pass@1 | Problems | Status |
-| :---- | ----: | ----: | :---- |
-| **MiniMax-M2-BF16** | **35.71%** | 182 | ✅ Complete |
-| **MiniMax-M2-THRIFT** | 🔄 Coming Soon | 182 | ⏳ Not Started Yet |
+| Model                 |        pass@1 | Problems | Status               |
+| :-------------------- | ------------: | -------: | :------------------- |
+| **MiniMax-M2-BF16**   |    **35.71%** |      182 | ✅ Complete           |
+| **MiniMax-M2-THRIFT** |    **36.81%** |      182 | ✅ Complete           |
+| **Δ (Difference)**    | **+1.10% ⬆️** |        - | **THRIFT Better!** ✨ |
 
 ---
 
-## 📈 Analysis (Preliminary)
-
-### Key Findings
-
-**MMLU Performance Drop**
-
-* THRIFT-BF16 shows **\-5.44%** overall MMLU drop  
-* Largest drop: **arc\_challenge (-12.20%)**  
-* Smallest drop: **winogrande (-1.58%)**  
-* **RTE improved by \+4.69%** 🎉
+## 📈 Analysis (Updated)
 
-**Subject-Specific Performance**
+**Highlights**
 
-* Best preservation: **Social Sciences (-3.18%)**  
-* Most degraded: **Other (-7.02%)**  
-* STEM: **Moderate drop (-3.30%)**
+* **THRIFT wins** on **GSM8K (+0.53%)** and **LiveCodeBench (+1.10%)**, and on **RTE (+4.69%)**.
+* **BF16 leads** on broad **MMLU**, **HumanEval**, **MBPP**, and tasks like **arc_challenge**.
 
 **Compression Trade-off**
 
-* THRIFT-BF16 (compressed) vs BF16 (base)  
-* Average accuracy loss: **\~4–5%**  
-* Expected for compressed/quantized models
+* Average knowledge-task drop for THRIFT is ~**4–5%**, with **math preserved or slightly improved**.
 
-**MMLU Category Breakdown**
+**Subject Breakdown (MMLU)**
 
-| Category | BF16 (Base) | THRIFT-BF16 | Difference | Status |
-| :---- | ----: | ----: | ----: | :---- |
-| High School Government | 97.93% | 94.82% | \-3.11% | ✅ Still Excellent |
-| High School Psychology | 95.41% | 93.58% | \-1.83% | ✅ Well Preserved |
-| Marketing | 95.73% | 91.88% | \-3.85% | ✅ Good |
-| Professional Medicine | 92.28% | 79.78% | \-12.50% | ⚠️ Notable Drop |
-| Clinical Knowledge | 92.83% | 85.66% | \-7.17% | ⚠️ Moderate Drop |
+| Category               | BF16 (Base) | THRIFT-BF16 | Difference | Status            |
+| :--------------------- | ----------: | ----------: | ---------: | :---------------- |
+| High School Government |      97.93% |      94.82% |     -3.11% | ✅ Still Excellent |
+| High School Psychology |      95.41% |      93.58% |     -1.83% | ✅ Well Preserved  |
+| Marketing              |      95.73% |      91.88% |     -3.85% | ✅ Good            |
+| Professional Medicine  |      92.28% |      79.78% |    -12.50% | ⚠️ Notable Drop   |
+| Clinical Knowledge     |      92.83% |      85.66% |     -7.17% | ⚠️ Moderate Drop  |
 
 ---
+
 ## **sglang Deployment with Python**
 
 It is recommended to use a virtual environment (such as **venv**, **conda**, or **uv**) to avoid dependency conflicts.
@@ -221,9 +213,10 @@ curl http://localhost:8000/v1/chat/completions \
         ]
     }'
 ```
+
 ## Benchmarks
 
-Coming soon.
+See the tables above for the latest **MMLU**, **MBPP**, **HumanEval**, **GSM8K**, **MATH-500**, and **LiveCodeBench** results (updated **November 9, 2025**).
 
 ## Research paper
 
@@ -267,7 +260,7 @@ Model conversion and HF Transformers code by @Qubitum at ModelCloud.
 
 @article{yang2025wanda++,
   title        = {Wanda++: Pruning Large Language Models via Regional Gradients},
-  author       = {Yang, Yifan and Zhen, Kai and Ganesh, Bhavana and Galstyan, Aram and Huybrechts, Goeric and M{"u}ller, Markus and K{"u}bler, Jonas M. and Swaminathan, Rupak Vignesh and Mouchtaris, Athanasios and Bodapati, Sravan Babu and Susanj, Nathan and Zhang, Zheng and FitzGerald, Jack and Kumar, Abhishek},
+  author       = {Yang, Yifan and Zhen, Kai and Ganesh, Bhavana and Galstyan, Aram and Huybrechts, Goeric and Müller, Markus and Kübler, Jonas M. and Swaminathan, Rupak Vignesh and Mouchtaris, Athanasios and Bodapati, Sravan Babu and Susanj, Nathan and Zhang, Zheng and FitzGerald, Jack and Kumar, Abhishek},
   journal      = {arXiv preprint arXiv:2503.04992},
   year         = {2025},
   eprinttype   = {arXiv},
@@ -307,7 +300,7 @@ Model conversion and HF Transformers code by @Qubitum at ModelCloud.
 
 @article{yang2023wanda,
   title        = {Wanda: Pruning by Weights and Activation-based Discriminant Analysis},
-  author       = {Yang, Yifan and Ganesh, Bhavana and Galstyan, Aram and Huybrechts, Goeric and M{"u}ller, Markus and Kübler, Jonas M. and Swaminathan, Rupak Vignesh and Mouchtaris, Athanasios and Bodapati, Sravan Babu and Zhang, Zheng and FitzGerald, Jack and Kumar, Abhishek},
+  author       = {Yang, Yifan and Ganesh, Bhavana and Galstyan, Aram and Huybrechts, Goeric and Müller, Markus and Kübler, Jonas M. and Swaminathan, Rupak Vignesh and Mouchtaris, Athanasios and Bodapati, Sravan Babu and Zhang, Zheng and FitzGerald, Jack and Kumar, Abhishek},
   journal      = {arXiv preprint arXiv:2306.11695},
   year         = {2023},
   eprinttype   = {arXiv},