README.md

---
library_name: transformers
pipeline_tag: text-generation
language:
  - en
license: mit
datasets:
  - roneneldan/TinyStories
tags:
  - qwen3-next
  - gated-deltanet
  - hybrid-attention
  - mixture-of-experts
  - moe
  - tinystories
  - tiny-model
  - text-generation
  - validation
  - debug-model
---

# Tiny Qwen3-Next 3M

This repository contains a tiny `Qwen3NextForCausalLM` Mixture-of-Experts
language model trained from scratch on TinyStories.

The model has **2,945,914 parameters**. It combines Gated DeltaNet linear
attention, gated full attention, sparse top-2 MoE routing, and a shared expert
in a checkpoint small enough for implementation testing and experimentation.

This is a synthetic tiny checkpoint. It is not an official Qwen model, does
not contain weights from an original Qwen checkpoint, and should not be
expected to match the quality or capabilities of production Qwen models.

## Repository contents

- `hf/`: the final Hugging Face checkpoint and tokenizer
- `example_generate.py`: a minimal generation example
- `eval_text_generation.json`: generations from the final training evaluation
- `artifact_metadata.json`: training arguments, metrics, router usage, and the
  expanded configuration
- `qwen3_next_config_dump.json`: a standalone configuration dump

Optimizer checkpoints and the full training log are intentionally omitted
from the distribution package.

## Important architecture distinction

This checkpoint uses:

```text
Qwen3NextConfig
Qwen3NextForCausalLM
model_type: qwen3_next
```

It does **not** use the separate regular Qwen3 MoE implementation:

```text
Qwen3MoeConfig
Qwen3MoeForCausalLM
model_type: qwen3_moe
```

The model follows the Qwen3-Next hybrid layer pattern: three Gated DeltaNet
linear-attention layers followed by one gated full-attention layer. Every
layer contains a sparse MoE block.

## Relationship to Qwen3-Next 80B

The architecture is a deliberately scaled-down member of the same
Transformers architecture family as `Qwen/Qwen3-Next-80B-A3B-Instruct`:

```text
3 x (Gated DeltaNet -> MoE)
1 x (Gated full attention -> MoE)
```

The original model repeats this four-layer pattern multiple times and uses
hundreds of experts. Tiny Qwen3-Next 3M keeps one four-layer cycle and eight
experts so that the important execution paths remain present at a much
smaller scale.

This checkpoint is pretrained only. It has not been instruction-tuned and
does not reproduce Qwen3-Next 80B behavior.

## Model architecture

```yaml
architecture: Qwen3NextForCausalLM
model_type: qwen3_next
parameter_count: 2,945,914

model_vocab_size: 1,024
tokenizer_size: 1,003
hidden_size: 216
intermediate_size: 540
num_hidden_layers: 4

layer_types:
  - linear_attention
  - linear_attention
  - linear_attention
  - full_attention

num_attention_heads: 8
num_key_value_heads: 1
head_dim: 56
partial_rotary_factor: 0.25
rope_theta: 10,000,000

linear_num_key_heads: 4
linear_key_head_dim: 54
linear_num_value_heads: 8
linear_value_head_dim: 54
linear_conv_kernel_dim: 4

num_experts: 8
num_experts_per_tok: 2
moe_intermediate_size: 54
shared_expert_intermediate_size: 54
norm_topk_prob: true
router_aux_loss_coef: 0.01

tie_word_embeddings: true
rms_norm_eps: 1.0e-6
max_position_embeddings: 1,024
```

The Gated DeltaNet dimensions retain the characteristic Qwen3-Next ratios:

- key width: `4 x 54 = 216`, equal to the hidden size
- value width: `8 x 54 = 432`, twice the hidden size

Full attention uses eight query heads and one key/value head. The query width
is `8 x 56 = 448`, while the key/value width is 56.

## MoE configuration

Each of the four decoder layers contains eight routed experts and one shared
expert. Every token selects two of the eight routed experts:

```yaml
routed_experts: 8
top_k: 2
shared_expert: true
normalized_top_k_weights: true
```

Router auxiliary loss was active during training. All experts received
traffic. Final aggregate expert fractions ranged from approximately 0.060 to
0.208 depending on layer and expert; no expert was unused.

## Training data

The model was trained on the full TinyStories training corpus using an
independent 1% validation split:

```yaml
selected_stories: 2,119,489
training_stories: 2,098,294
validation_stories: 21,195
validation_fraction: 0.01

training_blocks: 2,441,089
validation_blocks: 24,608
block_size: 256
```

Stories were joined into a continuous packed stream:

```text
BOS + story 1 + EOS + BOS + story 2 + EOS + ...
```

The stream was split into fixed 256-token blocks without padding. Only the
final incomplete block was discarded.

## Tokenizer

The checkpoint uses a small custom byte-level BPE tokenizer. The base BPE
vocabulary was trained with:

```text
BPE()
ByteLevel(add_prefix_space=False)
base_vocab_size: 1,000
min_frequency: 2
normalizer: None
```

Special tokens were then added at fixed IDs:

```text
<s>          -> 1000
</s>         -> 1001
<|im_start|> -> 1002
```

The tokenizer uses `<s>` as both BOS and padding, and `</s>` as EOS. The model
configuration reserves 1,024 embedding rows while the tokenizer exposes
1,003 tokens.

## Training setup

The checkpoint was trained from scratch in float32 on an NVIDIA GeForce GTX
1650:

```yaml
dtype: float32
batch_size: 16
block_size: 256
training_steps: 152,568
epochs: 1.0
tokens_processed: 624,918,528

optimizer: AdamW
learning_rate: 3.0e-4
warmup_steps: 1,000
scheduler: warmup + cosine decay
minimum_learning_rate: 3.0e-5
weight_decay: 0.0
grad_clip: 1.0

training_time: 13h 48m 45s
```

## Evaluation

The final checkpoint produced:

```yaml
final_train_loss: 1.4145
validation_loss: 1.4320
validation_perplexity: 4.1871
```

Validation loss was computed over 16 batches, or 65,536 tokens, from the
independent packed validation split. These numbers are compact checkpoint
diagnostics, not general language-model benchmark results.

## Example generation

With prompt `Once upon`, sampling seed 0 produced the following representative
TinyStories-style opening:

```text
Once upon a time, there was a little girl named Lily. She loved to play with
her toys and go on adventures. One day, she went outside and saw a rainbow in
the sky. It was so pretty that it made her feel even happier than before.
```

Sampling is stochastic. Other seeds may produce a boy, an adult, an animal,
or another kind of TinyStories character. The model usually produces
recognizable English, but semantic contradictions, unfinished sentences,
invented words, and repetition remain possible at this size.

## Usage

Install the requirements:

```bash
pip install -r requirements.txt
```

Run the included local example from the repository root:

```bash
python example_generate.py
```

To load the package from Hugging Face Hub, resolve the `hf` directory to a
local path first. This also avoids a Transformers 5.14.1 local-subfolder issue
in which generation configuration lookup may incorrectly fall back to the
repository root:

```python
import torch
from pathlib import Path
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "shibatch/tinyqwen3next3m"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_dir = Path(snapshot_download(
    repo_id=repo,
    allow_patterns=["hf/*"],
)) / "hf"

tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    dtype=torch.float32,
    experts_implementation="batched_mm",
).to(device)
model.eval()

prompt = "Once upon"
input_ids = torch.tensor(
    [[tokenizer.bos_token_id] + tokenizer.encode(
        prompt,
        add_special_tokens=False,
    )],
    dtype=torch.long,
    device=device,
)

torch.manual_seed(0)
if device.type == "cuda":
    torch.cuda.manual_seed_all(0)

with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        top_k=40,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
        output_router_logits=False,
    )

print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))
```

## Loading requirements

The checkpoint requires a Transformers release containing
`Qwen3NextForCausalLM`. It was trained and tested with:

```text
transformers 5.14.1
torch 2.14.0.dev20260720+cu126
```

The optional `flash-linear-attention` and `causal-conv1d` packages are not
required. Without them, Transformers uses its PyTorch Gated DeltaNet fallback,
which is the path used to train and validate this checkpoint.

The examples explicitly select `experts_implementation="batched_mm"`. The
model's routed experts have an intermediate width of 54, while the default
PyTorch `grouped_mm` CUDA path in some recent Torch/Transformers combinations
requires expert matrix strides to be multiples of 16 bytes. Without the
explicit compatible implementation, loading succeeds but the first forward
pass can fail with:

```text
RuntimeError: strides should be multiple of 16 bytes
```

`batched_mm` evaluates the same expert weights without that grouped-kernel
layout restriction. `experts_implementation="eager"` is also a compatible,
slower fallback.

## Intended uses

This model is intended for:

- testing `Qwen3NextConfig` and `Qwen3NextForCausalLM`
- testing Gated DeltaNet fallback implementations
- testing the hybrid linear/full-attention layer pattern
- testing sparse top-2 MoE routing and shared experts
- checking router load-balancing loss
- exercising custom tokenizer loading
- testing `generate()`, `save_pretrained()`, and `from_pretrained()`
- compact inference-engine and architecture experiments

It is not intended for:

- instruction following or chat
- factual question answering
- high-quality long-form generation
- production deployment
- safety-critical use
- benchmark comparison with production Qwen models

## Limitations

Known limitations include:

- only 2.95 million parameters
- small 1,003-token tokenizer
- English TinyStories-only pretraining
- weak factual knowledge and reasoning
- no instruction tuning or chat template
- occasional grammatical and semantic errors
- invented words and truncated sentences
- repetition and template-like stories
- no quality-equivalence claim with official Qwen models
- no current llama.cpp or GGUF inference support assumed for Qwen3-Next

## Notes on GGUF

The checkpoint is distributed as a normal float32 Hugging Face Safetensors
model. A useful GGUF build requires converter and runtime support for the full
Qwen3-Next graph, including Gated DeltaNet, hybrid attention, routed experts,
and the shared expert. Merely placing tensors in a GGUF container is not
sufficient for compatible inference.

## Citation

This is a synthetic tiny Qwen3-Next-compatible MoE checkpoint trained from
scratch on TinyStories. It is intended for implementation validation,
debugging, education, and small-scale architecture experiments.