Xylolabs

Cosmos3-Nano-Text GGUF

Base model Formats License Code

GGUF quants of nvidia/Cosmos3-Nano for llama.cpp, on CUDA, Metal or CPU.

Files

  • Cosmos3-Nano-Text-<QUANT>.gguf is one file per quant.
  • Contents: Each file holds the full language model plus the original tokenizer and chat template.

-imat in a file name marks a quant made with an importance matrix.

Quant Size Mean KL vs BF16 Same top-1 vs BF16
Q8_0 8.71 GB 0.0013 97.84%
Q6_K 6.73 GB 0.0044 96.10%
Q5_K_M 5.85 GB 0.0090 94.79%
IQ4_XS 4.59 GB 0.0386 90.27%
IQ4_XS-imat 4.56 GB 0.0222 92.26%

The reference is our BF16 pass in transformers. KL and top-1 cover 16,272 positions of English and Korean text.

Mean KL and same top-1 against BF16 by file size

Tensor types

Quant Tensor types and counts
IQ4_XS-imat F32: 145, IQ4_XS: 217, Q5_K: 36, Q6_K: 1
IQ4_XS F32: 145, IQ4_XS: 213, Q5_K: 40, Q6_K: 1
Q5_K_M F32: 145, Q5_K: 217, Q6_K: 37
Q6_K F32: 145, Q6_K: 254
Q8_0 F32: 145, Q8_0: 254

How to run

Use llama.cpp commit cd959bae99ff4bed22bef2604f45241c09bbb8c8 or later.

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout cd959bae99ff4bed22bef2604f45241c09bbb8c8
cmake -B build -DCMAKE_BUILD_TYPE=Release        # add -DGGML_CUDA=ON for NVIDIA GPUs
cmake --build build -j --target llama-cli llama-server

./build/bin/llama-cli -m Cosmos3-Nano-Text-Q8_0.gguf --jinja -c 8192 -ngl 99

llama-server takes the same flags.

License and notices

OpenMDW-1.1, copied in LICENSE: NVIDIA's license for Cosmos3-Nano. The model starts from Qwen3-VL-8B, so the Qwen Team's Apache License 2.0 is kept too, in APACHE_LICENSE. We extracted the text-understanding tower into Qwen3-compatible tensor names, converted it to GGUF and quantized it. NOTICE names NVIDIA and the Qwen Team and links the sources. Xylolabs is not affiliated with or endorsed by NVIDIA, Qwen or Alibaba Cloud.

Citation

Cite the original Cosmos3-Nano model card and its linked Cosmos 3 technical report.

Downloads last month
1,251
GGUF
Model size
8B params
Architecture
qwen3
Hardware compatibility
Log In to add your hardware

4-bit

5-bit

6-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Xylolabs/Cosmos3-Nano-Text-GGUF

Quantized
(14)
this model

Collection including Xylolabs/Cosmos3-Nano-Text-GGUF