Update README.md
Browse files
README.md
CHANGED
|
@@ -20,7 +20,6 @@ base_model: sarvamai/sarvam-105b
|
|
| 20 |
|
| 21 |
This model is quantized using llm-compressor.
|
| 22 |
Calibration dataset [sarvamai/indivibe](https://huggingface.co/datasets/sarvamai/indivibe)
|
| 23 |
-
Benchmarks compared to bf16 are provided.
|
| 24 |
|
| 25 |
|
| 26 |
## Deployment
|
|
@@ -56,28 +55,10 @@ vllm serve
|
|
| 56 |
## Model Files
|
| 57 |
| File Size | Last Updated |
|
| 58 |
|-----------|--------------|
|
| 59 |
-
| `
|
| 60 |
|
| 61 |
## Logs
|
| 62 |
```
|
| 63 |
2026-03-12
|
| 64 |
1. Initial commit
|
| 65 |
```
|
| 66 |
-
|
| 67 |
-
## Evaluation
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
Benchmark | Metric | Config | **BF16 (Original)** | **AWQ (4-bit)** | Diff | Recovery |
|
| 71 |
-
| :--- | :--- | :--- | :---: | :---: | :---: | :---: |
|
| 72 |
-
| **BBH** | exact_match | 3-shot | **63.22%** | 58.61% | 🔻 -4.61% | 92.7% |
|
| 73 |
-
| **GSM8K** | strict-match | 5-shot (Direct) | **72.40%** | 63.91% | 🔻 -8.49% | 88.3% |
|
| 74 |
-
| **GSM8K** | flexible-extract | 5-shot (Direct) | **69.90%** | 55.72% | 🔻 -14.18% | 79.7% |
|
| 75 |
-
| **GSM8K (CoT)** | strict-match | 8-shot (CoT) | **72.71%** | 76.80% | 🔺 +4.09% | 105.6% |
|
| 76 |
-
| **GSM8K (CoT)** | flexible-extract | 8-shot (CoT) | **82.41%** | 80.14% | 🔻 -2.27% | 97.2% |
|
| 77 |
-
| **MMLU** | acc | ⚠️ 0-shot | **43.40%** | 43.86% | 🔺 +0.46% | 101.1% |
|
| 78 |
-
| **ARC-Challenge** | acc | ⚠️ 0-shot | **29.10%** | 26.96% | 🔻 -2.14% | 92.6% |
|
| 79 |
-
| **HellaSwag** | acc | ⚠️ 0-shot | **40.67%** | 40.29% | 🔻 -0.38% | 99.1% |
|
| 80 |
-
| **HellaSwag** | acc_norm | ⚠️ 0-shot | **51.75%** | 50.27% | 🔻 -1.48% | 97.1% |
|
| 81 |
-
| **IFEval** | inst_level_strict | 0-shot | **32.85%** | 32.13% | 🔻 -0.72% | 97.8% |
|
| 82 |
-
| **TruthfulQA MC2** | acc | 0-shot | **49.71%** | 50.75% | 🔺 +1.04% | 102.1% |
|
| 83 |
-
| **Winogrande** | acc | ⚠️ 0-shot | **51.14%** | 49.49% | 🔻 -1.65% | 96.8% |
|
|
|
|
| 20 |
|
| 21 |
This model is quantized using llm-compressor.
|
| 22 |
Calibration dataset [sarvamai/indivibe](https://huggingface.co/datasets/sarvamai/indivibe)
|
|
|
|
| 23 |
|
| 24 |
|
| 25 |
## Deployment
|
|
|
|
| 55 |
## Model Files
|
| 56 |
| File Size | Last Updated |
|
| 57 |
|-----------|--------------|
|
| 58 |
+
| `74GiB` | `2026-03-12` |
|
| 59 |
|
| 60 |
## Logs
|
| 61 |
```
|
| 62 |
2026-03-12
|
| 63 |
1. Initial commit
|
| 64 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|