JunHowie commited on
Commit
851e587
·
verified ·
1 Parent(s): 592444d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -20
README.md CHANGED
@@ -20,7 +20,6 @@ base_model: sarvamai/sarvam-105b
20
 
21
  This model is quantized using llm-compressor.
22
  Calibration dataset [sarvamai/indivibe](https://huggingface.co/datasets/sarvamai/indivibe)
23
- Benchmarks compared to bf16 are provided.
24
 
25
 
26
  ## Deployment
@@ -56,28 +55,10 @@ vllm serve
56
  ## Model Files
57
  | File Size | Last Updated |
58
  |-----------|--------------|
59
- | `26GiB` | `2026-03-12` |
60
 
61
  ## Logs
62
  ```
63
  2026-03-12
64
  1. Initial commit
65
  ```
66
-
67
- ## Evaluation
68
-
69
-
70
- Benchmark | Metric | Config | **BF16 (Original)** | **AWQ (4-bit)** | Diff | Recovery |
71
- | :--- | :--- | :--- | :---: | :---: | :---: | :---: |
72
- | **BBH** | exact_match | 3-shot | **63.22%** | 58.61% | 🔻 -4.61% | 92.7% |
73
- | **GSM8K** | strict-match | 5-shot (Direct) | **72.40%** | 63.91% | 🔻 -8.49% | 88.3% |
74
- | **GSM8K** | flexible-extract | 5-shot (Direct) | **69.90%** | 55.72% | 🔻 -14.18% | 79.7% |
75
- | **GSM8K (CoT)** | strict-match | 8-shot (CoT) | **72.71%** | 76.80% | 🔺 +4.09% | 105.6% |
76
- | **GSM8K (CoT)** | flexible-extract | 8-shot (CoT) | **82.41%** | 80.14% | 🔻 -2.27% | 97.2% |
77
- | **MMLU** | acc | ⚠️ 0-shot | **43.40%** | 43.86% | 🔺 +0.46% | 101.1% |
78
- | **ARC-Challenge** | acc | ⚠️ 0-shot | **29.10%** | 26.96% | 🔻 -2.14% | 92.6% |
79
- | **HellaSwag** | acc | ⚠️ 0-shot | **40.67%** | 40.29% | 🔻 -0.38% | 99.1% |
80
- | **HellaSwag** | acc_norm | ⚠️ 0-shot | **51.75%** | 50.27% | 🔻 -1.48% | 97.1% |
81
- | **IFEval** | inst_level_strict | 0-shot | **32.85%** | 32.13% | 🔻 -0.72% | 97.8% |
82
- | **TruthfulQA MC2** | acc | 0-shot | **49.71%** | 50.75% | 🔺 +1.04% | 102.1% |
83
- | **Winogrande** | acc | ⚠️ 0-shot | **51.14%** | 49.49% | 🔻 -1.65% | 96.8% |
 
20
 
21
  This model is quantized using llm-compressor.
22
  Calibration dataset [sarvamai/indivibe](https://huggingface.co/datasets/sarvamai/indivibe)
 
23
 
24
 
25
  ## Deployment
 
55
  ## Model Files
56
  | File Size | Last Updated |
57
  |-----------|--------------|
58
+ | `74GiB` | `2026-03-12` |
59
 
60
  ## Logs
61
  ```
62
  2026-03-12
63
  1. Initial commit
64
  ```