tiny-d3-nano-for-testing-GGUF

This is a modified, damaged copy of vllm-sr/d3-nano, made only to test software. Its output is gibberish. Do not use it for anything else.

GGUF of ggml-org-dev/tiny-d3-nano-for-testing, converted with convert_hf_to_gguf.py of llama.cpp (text model and mmproj in F16). Used by the llama.cpp server tests of the vllm-d3 decision type.

Changes on top of the HF slice:

  • The vocab of the text model is cut from 248320 tokens to 3348 (slice_vocab_gguf.py of the llama.cpp server tests: special tokens, all single characters, the BPE path of the decision label words (option codes A..Z, AA..ZZ), and the tokens of the first 2000 merges are kept; ids are renumbered and the vocab-sized rows of the token embedding and of the LM head built from the readout are sliced to match). Text is still tokenized correctly, but into more, shorter tokens than with the original vocab.
  • The text model is quantized to Q4_0 with llama-quantize (token embedding and output included), the mmproj to Q8_0.
llama-server -hf ggml-org-dev/tiny-d3-nano-for-testing-GGUF:Q4_0

License

Derived from d3-nano by the vLLM Semantic Router Team (itself built on Qwen/Qwen3.5-2B), under the Apache License 2.0. See NOTICE.

Downloads last month
-
GGUF
Model size
438k params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ggml-org-dev/tiny-d3-nano-for-testing-GGUF

Finetuned
Qwen/Qwen3.5-2B
Finetuned
vllm-sr/d3-nano
Quantized
(1)
this model