ngxson's picture
ngxson HF Staff
tiny sliced copy for testing
8b28669 verified
|
Raw History Blame Contribute Delete
1.51 kB
metadata
license: apache-2.0
base_model: Qwen/Qwen3.5-4B-Base
tags:
  - testing

tiny-Qwen3.5-4B-Base-for-testing

This is a modified, damaged copy of Qwen/Qwen3.5-4B-Base (revision 1001bb4), made only to test software. Its output is gibberish. Do not use it for anything else.

It keeps the architecture (hybrid linear attention / gated full attention, MTP layer, vision encoder), tokenizer and configs of Qwen3.5-4B-Base, but it is small enough for CI. It is the base model of the tiny LoRA decision adapter ggml-org-dev/tiny-kev-4b-for-testing: the llama.cpp converter downloads it and merges the adapter into it.

Changes from the original model

Sliced from the safetensors with slice_qwen35_lora.py, which follows the slice-model skill of llama.cpp:

  • Text model: 4 of 32 layers (0, 1, 2, 3: 3 linear attention + 1 full attention), hidden size 2560 -> 32, FFN 9216 -> 128, heads 16 -> 2, KV heads 4 -> 1 (head_dim 256 kept), linear attention key heads 16 -> 2, value heads 32 -> 6 (head dims 128 kept)
  • Token embedding: full vocabulary kept, narrowed to the hidden size (tied embeddings kept)
  • MTP layer: kept, narrowed the same way
  • Vision encoder: 24 -> 2 blocks, hidden size 1024 -> 128, heads 16 -> 2, FFN 4096 -> 256, output 2560 -> 32

License

Derived from Qwen3.5-4B-Base by the Qwen team, under the Apache License 2.0.