How to use from
llama.cpp
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ggml-org-dev/tiny-d3-nano-for-testing-GGUF:Q8_0
# Run inference directly in the terminal:
llama cli -hf ggml-org-dev/tiny-d3-nano-for-testing-GGUF:Q8_0
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ggml-org-dev/tiny-d3-nano-for-testing-GGUF:Q8_0
# Run inference directly in the terminal:
llama cli -hf ggml-org-dev/tiny-d3-nano-for-testing-GGUF:Q8_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf ggml-org-dev/tiny-d3-nano-for-testing-GGUF:Q8_0
# Run inference directly in the terminal:
./build/bin/llama cli -hf ggml-org-dev/tiny-d3-nano-for-testing-GGUF:Q8_0
Use Docker
docker model run hf.co/ggml-org-dev/tiny-d3-nano-for-testing-GGUF:Q8_0
Quick Links

tiny-d3-nano-for-testing-GGUF

This is a modified, damaged copy of vllm-sr/d3-nano, made only to test software. Its output is gibberish. Do not use it for anything else.

GGUF of ggml-org-dev/tiny-d3-nano-for-testing, converted with convert_hf_to_gguf.py of llama.cpp (text model and mmproj in F16). Used by the llama.cpp server tests of the vllm-d3 decision type.

Changes on top of the HF slice:

  • The vocab of the text model is cut from 248320 tokens to 3348 (slice_vocab_gguf.py of the llama.cpp server tests: special tokens, all single characters, the BPE path of the decision label words (option codes A..Z, AA..ZZ), and the tokens of the first 2000 merges are kept; ids are renumbered and the vocab-sized rows of the token embedding and of the LM head built from the readout are sliced to match). Text is still tokenized correctly, but into more, shorter tokens than with the original vocab.
  • The text model is quantized to Q4_0 with llama-quantize (token embedding and output included), the mmproj to Q8_0.
llama-server -hf ggml-org-dev/tiny-d3-nano-for-testing-GGUF:Q4_0

License

Derived from d3-nano by the vLLM Semantic Router Team (itself built on Qwen/Qwen3.5-2B), under the Apache License 2.0. See NOTICE.

Downloads last month
88
GGUF
Model size
438k params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ggml-org-dev/tiny-d3-nano-for-testing-GGUF

Finetuned
Qwen/Qwen3.5-2B
Finetuned
vllm-sr/d3-nano
Quantized
(1)
this model