Instructions to use r0b0tlab/XYZ-Aquila-mini-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use r0b0tlab/XYZ-Aquila-mini-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="r0b0tlab/XYZ-Aquila-mini-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("r0b0tlab/XYZ-Aquila-mini-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("r0b0tlab/XYZ-Aquila-mini-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use r0b0tlab/XYZ-Aquila-mini-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "r0b0tlab/XYZ-Aquila-mini-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "r0b0tlab/XYZ-Aquila-mini-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/r0b0tlab/XYZ-Aquila-mini-NVFP4

SGLang

How to use r0b0tlab/XYZ-Aquila-mini-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "r0b0tlab/XYZ-Aquila-mini-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "r0b0tlab/XYZ-Aquila-mini-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "r0b0tlab/XYZ-Aquila-mini-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "r0b0tlab/XYZ-Aquila-mini-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use r0b0tlab/XYZ-Aquila-mini-NVFP4 with Docker Model Runner:
```
docker model run hf.co/r0b0tlab/XYZ-Aquila-mini-NVFP4
```

XYZ-Aquila-mini NVFP4 (SM121)

True W4A4 NVFP4 (ModelOpt 0.45) of XYZAILab/XYZ-Aquila-mini @ 4d3b0c45663b8b40f9e6e57b64ff815c09c49904, served natively on NVIDIA GB10 / SM121 with vLLM 0.25.0.

Native model context: max_position_embeddings = 262144 (~256k).
Served window depends on host KV (max_model_len); NIAH uses a max-context protocol (25/50/90% of served max_model_len).

Runtime

docker pull ghcr.io/r0b0tlab/xyz-aquila-mini-nvfp4-vllm:v0.25.0-sm121-702f4814

docker run --gpus all --ipc=host --network host \
  -e PORT=18082 \
  -e MODEL_ID=/models/model \
  -e SERVED_MODEL_NAME=xyz-aquila-nvfp4 \
  -e QUANTIZATION=modelopt_mixed \
  -e KV_CACHE_DTYPE=fp8 \
  -e ATTENTION_BACKEND=flashinfer \
  -e MOE_BACKEND=flashinfer_b12x \
  -e LINEAR_BACKEND=flashinfer_cutlass \
  -e LANGUAGE_MODEL_ONLY=1 \
  -e GPU_MEMORY_UTILIZATION=0.85 \
  -e MAX_MODEL_LEN=32768 \
  -e MAX_NUM_SEQS=16 \
  -v $PWD:/models/model:ro \
  ghcr.io/r0b0tlab/xyz-aquila-mini-nvfp4-vllm:v0.25.0-sm121-702f4814

Long-context tip: raise MAX_MODEL_LEN toward 262144 with MAX_NUM_SEQS=1 and chunked prefill (MAX_NUM_BATCHED_TOKENS can stay 16384). Host-admit before claiming full native ctx.

Native markers: FlashInferCutlassNvFp4LinearKernel, FLASHINFER_B12X MoE, FlashInfer attn arch=sm121, FP8 KV. No Marlin/emulation.

Quant recipe

Eligible MLP/experts/lm_head: NVFP4 W4A4
Attention: FP8
KV cache at serve: FP8
Excluded: vision, routers, norms, embeddings, GDN aux, MTP (MTP off in derived config only)
Calibration: 512 windowed rows
Export: ~21 GiB, 3 shards; hf_quant 291 layers (NVFP4 161 / FP8 130)

Quality results

Suite	Score	Notes
BFCL v4 `multi_turn_base` (200)	64.00% campaign · 68.00% core-subset re-run	bfcl-eval==2025.12.17 · 0 infra errors
BFCL AST-600 micro	33.33% prior · 33.50% core-subset	multiple / parallel / parallel_multiple
AST multiple	35.0% / 35.5%	200 cases
AST parallel	46.5% / 48.5%	200 cases
AST parallel_multiple	18.5% / 16.5%	200 cases
API canaries	5/5 PASS	identity, zh math, JSON, tool, needle
NIAH MAX_CONTEXT_TEST (served M=native 262144)	3/3 PASS	depths 65,520 / 131,040 / 235,872 (25/50/90% of M−64)
NIAH (legacy served M=32768)	3/3 PASS	~8k/16k/29k — not full native capacity

Do not use diluted BFCL “Overall Acc” CSV columns when only a subset of categories was run.

Throughput (GB10, vllm bench serve)

1024 in / 256 out, ignore_eos, temp 0, 3 reps drop-first:

Conc.	Output tok/s	Total tok/s	mean TTFT ms	mean TPOT ms
1	64.6	325.4	694	12.8
2	82.4	415.3	2010	16.3
4	164.5	829.1	641	21.9
8	230.5	1162.1	1009	30.9
16	305.7	1540.9	1854	45.3

Conversion gates

Gate	Status
B1 BF16 CPU real-residency baseline (~70.2 GiB)	PASS
C W4A4 PTQ	PASS
D packed NVFP4 export audit	PASS
E SM121 native serve	PASS

Runtime repo: https://github.com/r0b0tlab/xyz-aquila-mini-nvfp4-sm121-vllm
Bench client: https://github.com/r0b0tlab/r0b0bench (core / core-subset profiles)
Evidence HTML: see runtime repo docs/evidence-report.html

License

Downloads last month: 76

Safetensors

Model size

19B params

Tensor type

BF16

F8_E4M3

Model tree for r0b0tlab/XYZ-Aquila-mini-NVFP4

Base model

Qwen/Qwen3.6-35B-A3B

Finetuned

XYZAILab/XYZ-Aquila-mini

Quantized

(3)

this model

r0b0tlab
/

XYZ-Aquila-mini-NVFP4