--- license: apache-2.0 base_model: logic65/Qwen3.8-Whittle-tri-14.7B tags: - qwen4_exp - mixture-of-experts - hyper-connections - per-layer-embeddings - n-gram-memory - model-compression - research ---

Whittle

# whittle-next — GGUF builds (research preview) **⚠️ Research artifact.** A 19.8B qwen4exp-architecture model built by weight surgery on [Whittle-tri-14.7B](https://huggingface.co/logic65/Qwen3.8-Whittle-tri-14.7B) (Qwen3.8-27B depth-compressed 64 -> 32 layers by parallel-compose merging, FFN width uncut, Apache-2.0), then repaired by SFT. It now **holds a conversation, follows the chat template, writes fenced code, and stops cleanly** — but it is factually thin and its arithmetic is approximate. Treat it as an architecture demonstrator, not an assistant. If this research is useful to you: **[ko-fi.com/davida81328](https://ko-fi.com/davida81328)** ☕ ## Serving settings — these are REQUIRED, not suggestions ```bash llama-server -m whittle-next-qwen4exp-sft-PLE4B-Q4_K_M.gguf -ngl 99 -c 8192 --jinja \ --override-kv qwen4exp.expert_used_count=int:104 ``` Request body: `temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05`. Two settings do almost all the work, and both were measured on this build: - **Sampling, not greedy.** At `temperature 0` a paragraph-length answer degenerates (4-gram repetition **0.885** — "the ocean is a combination of water and water…"). At the settings above the same prompt scores **0.000** and ends with `finish_reason=stop`. Greedy decoding is the single largest cause of looping in this model. - **k=104 active experts** (the `--override-kv` above). Raising k from the trained 58 to 104 is a config-only change that fixed list termination, restored task engagement (a "build a page" request went from a fabricated URL to real fenced HTML), and removed intra-list repetition — with zero gradient steps. Reasoning is optional: pass `chat_template_kwargs: {"enable_thinking": false}` for short factual turns. With thinking on, allow ≥700 tokens — the think block is verbose. ## The files | file | what it is | recommended | |---|---|---| | `whittle-next-qwen4exp-sft-PLE4B-Q4_K_M.gguf` | SFT + woken hyper-connections + trained shared-expert gates + 4B n-gram memory | ✅ **yes** | | `whittle-next-qwen4exp-router104-PLE4B-Q4_K_M.gguf` | the above **plus jointly-trained routers at k=104** — better offline metrics, worse behaviour (see below) | experimental | | `whittle-next-qwen4exp-HC-Q4_K_M.gguf` | hyper-connections only, no n-gram memory | ablation | | `whittle-next-qwen4exp-HC-PLE4B-f16.gguf` | f16, n-gram memory, pre-SFT | ablation | ## Measured behaviour (identical 6-probe harness, k=104, ai-2 / 3×RTX 3060) | build | mode | clean stops | looping answers | |---|---|---|---| | SFT (recommended) | thinking off | **5/6** | 1/6 | | SFT (recommended) | thinking on | 4/6 | 1/6 | | router104 | thinking off | 4/6 | 2/6 | | router104 | thinking on | 2/6 | — over-thinks, ran out of budget | With the recommended sampling the remaining loop disappears: longform, explanation, code and list probes all returned `finish_reason=stop` with 4-gram repetition 0.000 (one short story at 0.38). **Why router104 is not the default, despite better numbers.** Training the routers *jointly* with the shared-expert gates, hyper-connections and n-gram projections — and *at the k they serve* — produced the best offline metrics this project has recorded (held-out CE 4.1466 → **3.9745**, fact battery 4/5 → **5/5**). But served, it over-thinks and repeats more. The training-time gate was selecting on cross-entropy and a short greedy battery, neither of which measures paragraph-length generation; repetition on that gate rose 0.057 → 0.093 over the same window while CE improved. The router result is real and reproducible — it is a training-objective lesson, not a serving win. ## How conversational is it, really? Mechanically conversational; not yet substantively reliable. It takes a turn, answers, and stops — and the content underneath is often wrong. Verified single-turn probes (recommended build, k=104, serving sampler): Paris ✅, a complete valid fenced HTML page ✅, a coherent non-repeating paragraph ✅ — against "the sky is blue because sunlight shines through the clouds" ❌, 17+25 = 32 ❌, and "list exactly 5 fruits" sometimes answered "1, 2, 3, 4, and 5 are fruits" ❌ (it hears the format and misses the substance). **Untested:** every probe is single-turn. Multi-turn context retention — arguably the real test of "conversational" — has not been measured, and we make no claim about it. The failure mode has moved from *broken generation* to *a small model with damaged knowledge*. ## Honest limitations - **Facts are unreliable.** "Why is the sky blue" gets a confident wrong answer. The 4B n-gram memory measurably helps in-domain perplexity (86.6 → 33.2 on wikitext-2 at ctx 512) but does not make the model knowledgeable. - **Arithmetic is approximate**: 17 + 25 returns 32, 20 or 49 depending on build and mode. The model estimates where the parent computed. - **It will invent URLs and citations** rather than decline. Do not trust any link it produces. - Long code generations can still decay under greedy decoding; use the sampling settings. ## Architecture 19.775B parameters, 32 layers × 5120, 3:1 GDN:full-attention, 240 experts (k=104 recommended, 58 trained), 4 hyper-connection residual streams, per-layer n-gram memory over a 6.25M-row × 640 table (≈4B parameters, host-offloadable). Requires a llama.cpp with qwen4exp support; the GGUFs declare `output_gate_type: silu`, which transformers' `qwen4_exp` config now supports natively — the Qwen3.5-derived GDN weights need a SiLU output gate, not the sigmoid a Flash-Next model uses. **David Aylward (logic65)** & **Claude (Anthropic)** — designed, debugged and trained together.