| --- |
| license: apache-2.0 |
| base_model: nvidia/Agents-A1-FP4 |
| tags: |
| - modelopt |
| - nvfp4 |
| - fp4 |
| - gs16 |
| - compressed-tensors |
| - vllm |
| - moe |
| - mamba |
| - tensor-parallel |
| --- |
| |
| # Agents-A1-ModelOpt-NVFP4 |
|
|
| This repository contains an optimized **NVFP4 (Group Size 16)** quantized checkpoint of `nvidia/Agents-A1-FP4` (a 35B parameter hybrid Mamba-Attention Mixture-of-Experts agentic model) produced using **NVIDIA ModelOpt** and formatted for native **vLLM** serving. |
|
|
| ## Model Summary |
| * **Base Architecture**: Agents-A1 MoE (`Qwen3_5MoeForCausalLM` / hybrid Mamba-Attention) |
| * **Quantization Method**: NVIDIA ModelOpt NVFP4 (W4A16 per-group quantization) |
| * **Group Size**: **GS16** (Required configuration for vLLM Marlin FP4 CUDA kernel execution) |
| * **Total Model Size**: ~19.66 GB safetensors weight shards (6 shards) |
| * **Vocabulary Size**: 152,064 tokens (intact) |
| * **Vision Encoder & MTP**: Vision encoder intact (`visual.*`), vocabulary intact, MTP heads stripped (`mtp.*`) |
|
|
| ## Group Size & vLLM Serving Note |
| > **Critical Serving Note**: While Group Size 128 (GS128) shrinks footprint further, the vLLM Marlin FP4 CUDA kernel (`marlin_mm`) only supports a group size of 16. Attempts to serve GS128 will result in a engine crash (`Invalid thread config`). Therefore, **GS16 is the only viable serving configuration**. |
| |
| ## Measured Benchmark Results |
| * **GSM8K Math Reasoning (20 Representative Samples, temp=0.6, top_p=0.95)**: Evaluated under local test suite |
| * **ARC-Challenge Science Reasoning (20 Representative Samples, temp=0.6, top_p=0.95)**: Evaluated under local test suite |
| |
| ## Serving with vLLM |
| |
| ```bash |
| vllm serve Cadododoom/agents-a1-modelopt-nvfp4 \ |
| --served-model-name nvidia/Agents-A1-FP4 \ |
| --tensor-parallel-size 2 \ |
| --quantization compressed-tensors \ |
| --moe-backend marlin \ |
| --attention-backend flashinfer \ |
| --kv-cache-dtype fp8 \ |
| --max-model-len 112000 \ |
| --max-num-seqs 4 \ |
| --max-num-batched-tokens 4096 \ |
| --gpu-memory-utilization 0.96 \ |
| --enable-prefix-caching \ |
| --trust-remote-code \ |
| --host 0.0.0.0 \ |
| --port 30000 |
| ``` |
| |
