README.md

---
license: apache-2.0
base_model: logic65/Qwen3.8-Whittle-tri-14.7B
tags:
  - qwen4_exp
  - mixture-of-experts
  - hyper-connections
  - per-layer-embeddings
  - n-gram-memory
  - model-compression
  - research
---

<p align="center"><img src="whittle.svg" width="640" alt="Whittle"></p>

# whittle-next — GGUF builds (research preview)

**⚠️ Research artifact.** A 19.8B qwen4exp-architecture model built by weight surgery on
[Whittle-tri-14.7B](https://huggingface.co/logic65/Qwen3.8-Whittle-tri-14.7B) (Qwen3.8-27B depth-compressed
64 -> 32 layers by parallel-compose merging, FFN width uncut, Apache-2.0), then repaired by SFT. It now **holds a conversation, follows the chat
template, writes fenced code, and stops cleanly** — but it is factually thin and its arithmetic is
approximate. Treat it as an architecture demonstrator, not an assistant.

If this research is useful to you: **[ko-fi.com/davida81328](https://ko-fi.com/davida81328)** ☕

## Serving settings — these are REQUIRED, not suggestions

```bash
llama-server -m whittle-next-qwen4exp-sft-PLE4B-Q4_K_M.gguf -ngl 99 -c 8192 --jinja \
  --override-kv qwen4exp.expert_used_count=int:104
```
Request body: `temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05`.

Two settings do almost all the work, and both were measured on this build:

- **Sampling, not greedy.** At `temperature 0` a paragraph-length answer degenerates
  (4-gram repetition **0.885** — "the ocean is a combination of water and water…").
  At the settings above the same prompt scores **0.000** and ends with `finish_reason=stop`.
  Greedy decoding is the single largest cause of looping in this model.
- **k=104 active experts** (the `--override-kv` above). Raising k from the trained 58 to 104 is a
  config-only change that fixed list termination, restored task engagement (a "build a page" request
  went from a fabricated URL to real fenced HTML), and removed intra-list repetition — with zero
  gradient steps.

Reasoning is optional: pass `chat_template_kwargs: {"enable_thinking": false}` for short factual
turns. With thinking on, allow ≥700 tokens — the think block is verbose.

## The files

| file | what it is | recommended |
|---|---|---|
| `whittle-next-qwen4exp-sft-PLE4B-Q4_K_M.gguf` | SFT + woken hyper-connections + trained shared-expert gates + 4B n-gram memory | ✅ **yes** |
| `whittle-next-qwen4exp-router104-PLE4B-Q4_K_M.gguf` | the above **plus jointly-trained routers at k=104** — better offline metrics, worse behaviour (see below) | experimental |
| `whittle-next-qwen4exp-HC-Q4_K_M.gguf` | hyper-connections only, no n-gram memory | ablation |
| `whittle-next-qwen4exp-HC-PLE4B-f16.gguf` | f16, n-gram memory, pre-SFT | ablation |

## Measured behaviour (identical 6-probe harness, k=104, ai-2 / 3×RTX 3060)

| build | mode | clean stops | looping answers |
|---|---|---|---|
| SFT (recommended) | thinking off | **5/6** | 1/6 |
| SFT (recommended) | thinking on | 4/6 | 1/6 |
| router104 | thinking off | 4/6 | 2/6 |
| router104 | thinking on | 2/6 | — over-thinks, ran out of budget |

With the recommended sampling the remaining loop disappears: longform, explanation, code and list
probes all returned `finish_reason=stop` with 4-gram repetition 0.000 (one short story at 0.38).

**Why router104 is not the default, despite better numbers.** Training the routers *jointly* with
the shared-expert gates, hyper-connections and n-gram projections — and *at the k they serve* —
produced the best offline metrics this project has recorded (held-out CE 4.1466 → **3.9745**, fact
battery 4/5 → **5/5**). But served, it over-thinks and repeats more. The training-time gate was
selecting on cross-entropy and a short greedy battery, neither of which measures paragraph-length
generation; repetition on that gate rose 0.057 → 0.093 over the same window while CE improved. The
router result is real and reproducible — it is a training-objective lesson, not a serving win.

## How conversational is it, really?

Mechanically conversational; not yet substantively reliable. It takes a turn, answers, and stops —
and the content underneath is often wrong. Verified single-turn probes (recommended build, k=104,
serving sampler): Paris ✅, a complete valid fenced HTML page ✅, a coherent non-repeating paragraph ✅
— against "the sky is blue because sunlight shines through the clouds" ❌, 17+25 = 32 ❌, and
"list exactly 5 fruits" sometimes answered "1, 2, 3, 4, and 5 are fruits" ❌ (it hears the format and
misses the substance).

**Untested:** every probe is single-turn. Multi-turn context retention — arguably the real test of
"conversational" — has not been measured, and we make no claim about it.

The failure mode has moved from *broken generation* to *a small model with damaged knowledge*.

## Honest limitations

- **Facts are unreliable.** "Why is the sky blue" gets a confident wrong answer. The 4B n-gram
  memory measurably helps in-domain perplexity (86.6 → 33.2 on wikitext-2 at ctx 512) but does not
  make the model knowledgeable.
- **Arithmetic is approximate**: 17 + 25 returns 32, 20 or 49 depending on build and mode. The model
  estimates where the parent computed.
- **It will invent URLs and citations** rather than decline. Do not trust any link it produces.
- Long code generations can still decay under greedy decoding; use the sampling settings.

## Architecture

19.775B parameters, 32 layers × 5120, 3:1 GDN:full-attention, 240 experts (k=104 recommended,
58 trained), 4 hyper-connection residual streams, per-layer n-gram memory over a 6.25M-row × 640
table (≈4B parameters, host-offloadable). Requires a llama.cpp with qwen4exp support; the GGUFs
declare `output_gate_type: silu`, which transformers' `qwen4_exp` config now supports natively —
the Qwen3.5-derived GDN weights need a SiLU output gate, not the sigmoid a Flash-Next model uses.

**David Aylward (logic65)** & **Claude (Anthropic)** — designed, debugged and trained together.