| | --- |
| | license: mit |
| | base_model: LocoreMind/LocoOperator-4B |
| | tags: |
| | - nvfp4 |
| | - quantized |
| | - qwen3 |
| | - agent |
| | - tool-calling |
| | - code |
| | - nvidia |
| | - modelopt |
| | - spark |
| | pipeline_tag: text-generation |
| | --- |
| | |
| | # LocoOperator-4B — NVFP4 Quantized |
| |
|
| | NVFP4-quantized version of [LocoreMind/LocoOperator-4B](https://huggingface.co/LocoreMind/LocoOperator-4B), an agent/tool-calling model based on Qwen3-4B-Instruct. |
| |
|
| | ## Quantization Details |
| |
|
| | | Property | Value | |
| | |----------|-------| |
| | | **Base model** | LocoreMind/LocoOperator-4B (Qwen3-4B finetune) | |
| | | **Quantization** | NVFP4 (weights) + FP8 (KV cache) | |
| | | **Group size** | 16 | |
| | | **Tool** | NVIDIA TensorRT Model Optimizer (modelopt 0.35.0) | |
| | | **Calibration** | cnn_dailymail (default) | |
| | | **Original size** | ~8 GB (BF16) | |
| | | **Quantized size** | 2.7 GB | |
| | | **Excluded** | `lm_head` (kept in higher precision) | |
| |
|
| | ## Intended Use |
| |
|
| | Optimized for deployment on NVIDIA Blackwell GPUs (GB10/GB100), particularly the DGX Spark. The NVFP4 format leverages Blackwell's native FP4 tensor cores for maximum throughput. |
| |
|
| | Best suited for: |
| | - Agent/tool-calling workflows |
| | - Code generation |
| | - Instruction following |
| |
|
| | ## Usage |
| |
|
| | ### With transformers + modelopt |
| |
|
| | ```python |
| | from transformers import AutoModelForCausalLM, AutoTokenizer |
| | |
| | model = AutoModelForCausalLM.from_pretrained( |
| | "DJLougen/LocoOperator-4B-NVFP4", |
| | device_map="auto", |
| | torch_dtype="auto", |
| | ) |
| | tokenizer = AutoTokenizer.from_pretrained("DJLougen/LocoOperator-4B-NVFP4") |
| | ``` |
| |
|
| | ### With TensorRT-LLM |
| |
|
| | Convert to TensorRT-LLM engine for optimal inference performance on Spark/Blackwell hardware. |
| |
|
| | ## Quality Check |
| |
|
| | Example outputs (cnn_dailymail calibration text): |
| | |
| | **Before quantization:** |
| | > "I'm excited to be doing the final two films," he said. "I can't wait to see what happens." |
| | |
| | **After NVFP4 quantization:** |
| | > "I don't think I'll be particularly extravagant," Radcliffe said. "I don't think I'll be one of those people who, as soon as they turn 18, suddenly buy themselves a massive sports car collection or something similar." |
| | |
| | Both outputs are coherent and contextually appropriate. |
| | |
| | ## Hardware |
| | |
| | - **Quantized on:** NVIDIA DGX Spark (GB10, 128 GB unified memory) |
| | - **Docker image:** `nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev` |
| | - **Target deployment:** Any NVIDIA Blackwell GPU with FP4 tensor core support |
| | |
