tiny-h2o-lightning-4b-for-testing

This is a modified, damaged copy of h2oai/h2o-lightning-4b, made only to test software. Its output is gibberish. Do not use it for anything else.

It keeps the architecture, tokenizer, chat template, serve_config.json (prompt, labels, temperatures) and MTP layer of h2o-lightning-4b, so it runs the same code paths as the real model in llama.cpp (h2o-lightning decision type: labels read at the token after the Answer: prefill, labels and temperatures from serve_config.json, image input), but it is small enough for CI.

GGUF version: ggml-org-dev/tiny-h2o-lightning-4b-for-testing-GGUF

Changes from the original model

Sliced from the safetensors with slice_qwen35_decision.py (python3 slice_qwen35_decision.py h2oai/h2o-lightning-4b <out dir>), which follows the slice-model skill of llama.cpp:

  • Text model: 4 of 32 layers (0-3: 3 linear attention + 1 full attention), hidden size 2560 -> 32, FFN 9216 -> 64, heads 16 -> 2, KV heads 4 -> 1 (head_dim 256 kept), linear attention key heads 16 -> 2, value heads 32 -> 4 (head dims 128 kept). The MTP layer is narrowed the same way
  • Vision encoder: 2 of 24 blocks, hidden size 1024 -> 128, heads 16 -> 2 (head_dim 64 kept), FFN 4096 -> 256, output 2560 -> 32
  • h2o_lightning_shim.py is kept unchanged only because convert_hf_to_gguf.py detects the checkpoint by it
  • The other Python code, tests, label_rows.lightning (two-way mode), tokenizer/ copy, vllm_patches/, engine/ and assets of the original repo are not included

License

Derived from h2o-lightning-4b by H2O.ai (itself built on Qwen/Qwen3.5-4B), under the Apache License 2.0.

Downloads last month
-
Safetensors
Model size
9.36M params
Tensor type
F32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ggml-org-dev/tiny-h2o-lightning-4b-for-testing

Finetuned
Qwen/Qwen3.5-4B
Finetuned
(10)
this model
Quantizations
1 model