Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -15,7 +15,9 @@ tags:
|
|
| 15 |
|
| 16 |
NVFP4 (4-bit) quant of [CohereLabs/North-Mini-Code-1.0](https://huggingface.co/CohereLabs/North-Mini-Code-1.0), made with [llm-compressor](https://github.com/vllm-project/llm-compressor) (`nvfp4-pack-quantized`). ~17 GB. Experts quantized; router/gate/`lm_head` kept high-precision.
|
| 17 |
|
| 18 |
-
|
|
|
|
|
|
|
| 19 |
|
| 20 |
Serve with vLLM (needs `Cohere2MoeForCausalLM` support + `cohere_melody`):
|
| 21 |
|
|
|
|
| 15 |
|
| 16 |
NVFP4 (4-bit) quant of [CohereLabs/North-Mini-Code-1.0](https://huggingface.co/CohereLabs/North-Mini-Code-1.0), made with [llm-compressor](https://github.com/vllm-project/llm-compressor) (`nvfp4-pack-quantized`). ~17 GB. Experts quantized; router/gate/`lm_head` kept high-precision.
|
| 17 |
|
| 18 |
+
**Benchmarks (DGX Spark / GB10, vLLM):**
|
| 19 |
+
- HumanEval pass@1: **90.2%** — matches the FP8 build (90.2%), within one problem of bf16 (90.9%).
|
| 20 |
+
- Decode: **~57 tok/s** single-request (reasoning off) — **≈1.72×** the FP8 build, at ~17 GB vs ~28 GB.
|
| 21 |
|
| 22 |
Serve with vLLM (needs `Cohere2MoeForCausalLM` support + `cohere_melody`):
|
| 23 |
|