tiny-h2o-lightning-4b-for-testing-GGUF

This is a modified, damaged copy of h2oai/h2o-lightning-4b, made only to test software. Its output is gibberish. Do not use it for anything else.

GGUF of ggml-org-dev/tiny-h2o-lightning-4b-for-testing, converted with convert_hf_to_gguf.py of llama.cpp (text model and mmproj in F16). Used by the llama.cpp server tests of the h2o-lightning decision type.

Changes on top of the HF slice:

  • The vocab of the text model is cut from 248320 tokens to 3493 (slice_vocab_gguf.py of the llama.cpp server tests: special tokens, all single characters, the BPE path of the decision label words (the 255 labels of serve_config.json), and the tokens of the first 2000 merges are kept; ids are renumbered and the vocab-sized rows of the token embedding (tied with the output) are sliced to match). Text is still tokenized correctly, but into more, shorter tokens than with the original vocab.
  • The text model is quantized to Q4_0 with llama-quantize (token embedding and output included), the mmproj to Q8_0.
llama-server -hf ggml-org-dev/tiny-h2o-lightning-4b-for-testing-GGUF:Q4_0

License

Derived from h2o-lightning-4b by H2O.ai (itself built on Qwen/Qwen3.5-4B), under the Apache License 2.0.

Downloads last month
46
GGUF
Model size
487k params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ggml-org-dev/tiny-h2o-lightning-4b-for-testing-GGUF