| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - mobiquant |
| - quantization |
| - mixed-precision |
| - llama |
| --- |
| |
| # MoBiQuant Checkpoints |
|
|
| Calibrated OmniQuant and router parameters for **MoBiQuant** on four LLaMA base models. |
|
|
| **Code:** [SimWangArizona/MoBiQuant](https://github.com/SimWangArizona/MoBiQuant) |
|
|
| ## Available checkpoints |
|
|
| | Directory | Base model | |
| |---|---| |
| | `llama2-7b` | Llama-2-7B | |
| | `llama3-8b` | Llama-3-8B | |
| | `llama3-3b` | Llama-3.2-3B | |
| | `llama3-1b` | Llama-3.2-1B | |
|
|
| Each directory contains: |
|
|
| - `omni_parameters.pth`: learned quantization parameters. |
| - `router_parameters.pth`: learned router parameters. |
|
|
| These are supplemental parameter checkpoints. Original pretrained model weights and tokenizer files are required separately; this repository cannot be loaded directly with `AutoModelForCausalLM.from_pretrained()`. |
|
|
| ## Quantization configuration |
|
|
| - Weight-only quantization with 16-bit activations. |
| - Group size: **128**. |
| - Bit slices: **[2, 2, 2, 2]** — a 2-bit base plus three routed residual slices. |
| - Routing covers Attention q/k/v/o and MLP gate/up/down projections. |
| - Effective average bits depend on routing and the evaluation target; Attention is not fixed at 4 bits. |
|
|
| ## Download |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| |
| checkpoint_dir = snapshot_download( |
| repo_id="Simwang/MoBiQuant", |
| allow_patterns=["llama3-1b/*.pth"], |
| ) |
| ``` |
|
|
| Replace `llama3-1b` with another directory above to download its parameters. See the code repository for evaluation instructions. The current directory-resume workflow additionally expects `router_stats.pth`; that file is not included in this release, so the downloaded pair alone is not a complete directory-resume bundle. |
|
|
| ## Limitations |
|
|
| The implementation evaluates quantization using floating-point simulation. These files do not constitute a packed low-bit deployment format or establish inference speedups. |
|
|
| The Apache-2.0 metadata applies to this release; obtain and use the original base models under their respective model licenses and access conditions. |
|
|
