DeepseaOne commited on
Commit
11eb3dd
·
verified ·
1 Parent(s): 6e25ece

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +3 -1
README.md CHANGED
@@ -15,7 +15,9 @@ tags:
15
 
16
  NVFP4 (4-bit) quant of [CohereLabs/North-Mini-Code-1.0](https://huggingface.co/CohereLabs/North-Mini-Code-1.0), made with [llm-compressor](https://github.com/vllm-project/llm-compressor) (`nvfp4-pack-quantized`). ~17 GB. Experts quantized; router/gate/`lm_head` kept high-precision.
17
 
18
- > Note: HF auto-reports ~17B parameters because it counts the FP4-packed `uint8` tensors (2 weights per byte). The actual model is **30B total / 3B active** (MoE), same as the base model.
 
 
19
 
20
  Serve with vLLM (needs `Cohere2MoeForCausalLM` support + `cohere_melody`):
21
 
 
15
 
16
  NVFP4 (4-bit) quant of [CohereLabs/North-Mini-Code-1.0](https://huggingface.co/CohereLabs/North-Mini-Code-1.0), made with [llm-compressor](https://github.com/vllm-project/llm-compressor) (`nvfp4-pack-quantized`). ~17 GB. Experts quantized; router/gate/`lm_head` kept high-precision.
17
 
18
+ **Benchmarks (DGX Spark / GB10, vLLM):**
19
+ - HumanEval pass@1: **90.2%** — matches the FP8 build (90.2%), within one problem of bf16 (90.9%).
20
+ - Decode: **~57 tok/s** single-request (reasoning off) — **≈1.72×** the FP8 build, at ~17 GB vs ~28 GB.
21
 
22
  Serve with vLLM (needs `Cohere2MoeForCausalLM` support + `cohere_melody`):
23