Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -15,6 +15,8 @@ tags:
|
|
| 15 |
|
| 16 |
NVFP4 (4-bit) quant of [CohereLabs/North-Mini-Code-1.0](https://huggingface.co/CohereLabs/North-Mini-Code-1.0), made with [llm-compressor](https://github.com/vllm-project/llm-compressor) (`nvfp4-pack-quantized`). ~17 GB. Experts quantized; router/gate/`lm_head` kept high-precision.
|
| 17 |
|
|
|
|
|
|
|
| 18 |
Serve with vLLM (needs `Cohere2MoeForCausalLM` support + `cohere_melody`):
|
| 19 |
|
| 20 |
```bash
|
|
|
|
| 15 |
|
| 16 |
NVFP4 (4-bit) quant of [CohereLabs/North-Mini-Code-1.0](https://huggingface.co/CohereLabs/North-Mini-Code-1.0), made with [llm-compressor](https://github.com/vllm-project/llm-compressor) (`nvfp4-pack-quantized`). ~17 GB. Experts quantized; router/gate/`lm_head` kept high-precision.
|
| 17 |
|
| 18 |
+
> Note: HF auto-reports ~17B parameters because it counts the FP4-packed `uint8` tensors (2 weights per byte). The actual model is **30B total / 3B active** (MoE), same as the base model.
|
| 19 |
+
|
| 20 |
Serve with vLLM (needs `Cohere2MoeForCausalLM` support + `cohere_melody`):
|
| 21 |
|
| 22 |
```bash
|