File size: 2,052 Bytes
3f3a299 cf0ded8 3f3a299 cf0ded8 6c1de7d cf0ded8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | ---
license: apache-2.0
language:
- en
tags:
- mobiquant
- quantization
- mixed-precision
- llama
---
# MoBiQuant Checkpoints
Calibrated OmniQuant and router parameters for **MoBiQuant** on four LLaMA base models.
**Code:** [SimWangArizona/MoBiQuant](https://github.com/SimWangArizona/MoBiQuant)
## Available checkpoints
| Directory | Base model |
|---|---|
| `llama2-7b` | Llama-2-7B |
| `llama3-8b` | Llama-3-8B |
| `llama3-3b` | Llama-3.2-3B |
| `llama3-1b` | Llama-3.2-1B |
Each directory contains:
- `omni_parameters.pth`: learned quantization parameters.
- `router_parameters.pth`: learned router parameters.
These are supplemental parameter checkpoints. Original pretrained model weights and tokenizer files are required separately; this repository cannot be loaded directly with `AutoModelForCausalLM.from_pretrained()`.
## Quantization configuration
- Weight-only quantization with 16-bit activations.
- Group size: **128**.
- Bit slices: **[2, 2, 2, 2]** — a 2-bit base plus three routed residual slices.
- Routing covers Attention q/k/v/o and MLP gate/up/down projections.
- Effective average bits depend on routing and the evaluation target; Attention is not fixed at 4 bits.
## Download
```python
from huggingface_hub import snapshot_download
checkpoint_dir = snapshot_download(
repo_id="Simwang/MoBiQuant",
allow_patterns=["llama3-1b/*.pth"],
)
```
Replace `llama3-1b` with another directory above to download its parameters. See the code repository for evaluation instructions. The current directory-resume workflow additionally expects `router_stats.pth`; that file is not included in this release, so the downloaded pair alone is not a complete directory-resume bundle.
## Limitations
The implementation evaluates quantization using floating-point simulation. These files do not constitute a packed low-bit deployment format or establish inference speedups.
The Apache-2.0 metadata applies to this release; obtain and use the original base models under their respective model licenses and access conditions.
|