README.md

---
library_name: transformers
pipeline_tag: text-generation
language:
  - en
license: mit
datasets:
  - roneneldan/TinyStories
tags:
  - deepseek-v2
  - multi-head-latent-attention
  - mla
  - mixture-of-experts
  - moe
  - tinystories
  - tiny-model
  - text-generation
  - validation
  - debug-model
---

# Tiny DeepSeek V2 3M

This repository contains a tiny `DeepseekV2ForCausalLM` Mixture-of-Experts
language model trained from scratch on TinyStories.

The model has **2,928,392 parameters**. It combines DeepSeek V2-style
Multi-head Latent Attention (MLA), compressed key/value states, decoupled
RoPE, routed experts, and a shared expert in a compact checkpoint intended for
implementation testing and architecture experiments.

This is a synthetic tiny checkpoint. It is not an official DeepSeek model,
does not contain weights from an original DeepSeek checkpoint, and should not
be expected to match the quality or capabilities of production DeepSeek
models.

## Repository contents

- `hf/`: the final Hugging Face checkpoint and tokenizer
- `example_generate.py`: a minimal generation example
- `eval_text_generation.json`: generations from the final training evaluation
- `artifact_metadata.json`: training arguments, metrics, router usage, and the
  expanded configuration
- `deepseek_v2_config_dump.json`: a standalone configuration dump

Optimizer checkpoints and the full training history are intentionally omitted
from this distribution package.

## Architecture identity

This checkpoint uses:

```text
DeepseekV2Config
DeepseekV2ForCausalLM
model_type: deepseek_v2
```

It is specifically a DeepSeek V2 architecture, not DeepSeek V3 and not a
generic dense decoder renamed as DeepSeek. The checkpoint exercises the
DeepSeek V2 MLA and MoE implementations provided by Hugging Face Transformers.

## Model architecture

```yaml
architecture: DeepseekV2ForCausalLM
model_type: deepseek_v2
parameter_count: 2,928,392

model_vocab_size: 1,024
tokenizer_size: 1,003
hidden_size: 216
intermediate_size: 432
num_hidden_layers: 5

num_attention_heads: 8
num_key_value_heads: 8
qk_nope_head_dim: 16
qk_rope_head_dim: 16
v_head_dim: 32
q_lora_rank: null
kv_lora_rank: 64

first_k_dense_replace: 1
n_routed_experts: 4
n_shared_experts: 1
num_experts_per_tok: 1
moe_intermediate_size: 128
topk_method: greedy
norm_topk_prob: false
routed_scaling_factor: 4.0
aux_loss_alpha: 0.01
seq_aux: true

tie_word_embeddings: true
rms_norm_eps: 1.0e-6
max_position_embeddings: 2,048
rope_theta: 10,000
```

## Multi-head Latent Attention

Each attention head separates its query/key dimensions into:

```text
16 non-positional dimensions + 16 rotary dimensions
```

Values use 32 dimensions per head. The key/value path is compressed through a
64-dimensional latent projection (`kv_lora_rank: 64`) before being expanded
for attention. Query LoRA is disabled in this tiny configuration, while the
compressed KV path and decoupled rotary/non-rotary query-key components remain
active.

This keeps the checkpoint small while exercising the defining DeepSeek V2 MLA
code paths.

## Dense and MoE layers

The first decoder layer uses a dense MLP. The remaining four layers use
DeepSeek V2 MoE blocks:

```text
layer 0: dense MLP
layer 1: 4 routed experts, top-1 + 1 shared expert
layer 2: 4 routed experts, top-1 + 1 shared expert
layer 3: 4 routed experts, top-1 + 1 shared expert
layer 4: 4 routed experts, top-1 + 1 shared expert
```

Every token is sent to one routed expert, while the shared expert is evaluated
for all tokens. Sequence-level router auxiliary loss was enabled during
training.

All routed experts received traffic. Final aggregate routing fractions were
close to 25% for every expert:

| MoE layer | Expert 0 | Expert 1 | Expert 2 | Expert 3 |
| ---: | ---: | ---: | ---: | ---: |
| 1 | 0.2502 | 0.2487 | 0.2522 | 0.2488 |
| 2 | 0.2538 | 0.2481 | 0.2487 | 0.2495 |
| 3 | 0.2512 | 0.2454 | 0.2519 | 0.2514 |
| 4 | 0.2465 | 0.2535 | 0.2542 | 0.2458 |

## Training data

The model was trained on the full TinyStories training corpus using an
independent 1% validation split:

```yaml
selected_stories: 2,119,489
training_stories: 2,098,294
validation_stories: 21,195
validation_fraction: 0.01

training_blocks: 2,441,089
validation_blocks: 24,608
block_size: 256
```

Stories were joined into a continuous packed stream:

```text
BOS + story 1 + EOS + BOS + story 2 + EOS + ...
```

The stream was split into fixed 256-token blocks without padding. Only the
final incomplete block was discarded.

## Tokenizer

The checkpoint uses a small custom byte-level BPE tokenizer. The base BPE
vocabulary was trained with:

```text
BPE()
ByteLevel(add_prefix_space=False)
base_vocab_size: 1,000
min_frequency: 2
normalizer: None
```

Special tokens were then added at fixed IDs:

```text
<s>          -> 1000
</s>         -> 1001
<|im_start|> -> 1002
```

The tokenizer uses `<s>` as both BOS and padding, and `</s>` as EOS. The model
configuration reserves 1,024 embedding rows while the tokenizer exposes
1,003 tokens.

## Training setup

The checkpoint was trained from scratch in float32 on an NVIDIA GeForce GTX
1650:

```yaml
dtype: float32
batch_size: 16
block_size: 256
training_steps: 152,568
epochs: 1.0
tokens_processed: 624,918,528

optimizer: AdamW
learning_rate: 3.0e-4
warmup_steps: 2,000
scheduler: warmup + cosine decay
minimum_learning_rate: 3.0e-5
weight_decay: 0.0
grad_clip: 1.0
```

## Evaluation

The final checkpoint produced:

```yaml
final_train_loss: 1.4096
validation_loss: 1.4226
validation_perplexity: 4.1478
```

Validation loss was computed over 32 batches, or 131,072 tokens, from the
independent packed validation split. These values are compact checkpoint
diagnostics, not general language-model benchmark results.

## Example generation

One final evaluation sample for prompt `There was a little` was:

```text
There was a little girl named Lily who loved to explore. One day, she saw a
big bush with lots of yummy peaches on it. She wanted to eat one, but her mom
said no.

Lily didn't listen and kept trying to eat her peach.
```

Sampling is stochastic. The model usually produces recognizable
TinyStories-style English, but semantic contradictions, unfinished sentences,
invented words, and repetition remain possible at this size.

## Usage

Install the requirements:

```bash
pip install -r requirements.txt
```

Run the included local example from the repository root:

```bash
python example_generate.py
```

To load the package from Hugging Face Hub, resolve the `hf` directory to a
local path first. This avoids a Transformers 5.14.1 local-subfolder issue in
which generation configuration lookup may incorrectly fall back to the
repository root:

```python
from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "shibatch/tinydeepseekv2-3m"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_dir = Path(snapshot_download(
    repo_id=repo,
    allow_patterns=["hf/*"],
)) / "hf"

tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    dtype=torch.float32,
).to(device)
model.eval()

prompt = "There was a little"
input_ids = torch.tensor(
    [[tokenizer.bos_token_id] + tokenizer.encode(
        prompt,
        add_special_tokens=False,
    )],
    dtype=torch.long,
    device=device,
)

torch.manual_seed(0)
if device.type == "cuda":
    torch.cuda.manual_seed_all(0)

with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        top_k=40,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))
```

## Loading requirements

The checkpoint requires a Transformers release containing
`DeepseekV2ForCausalLM`. It was trained and tested with:

```text
transformers 5.14.1
torch 2.14.0.dev20260720+cu126
```

## Intended uses

This model is intended for:

- testing `DeepseekV2Config` and `DeepseekV2ForCausalLM`
- testing Multi-head Latent Attention and compressed KV states
- exercising decoupled rotary and non-rotary query/key dimensions
- testing dense-to-MoE layer transitions
- testing top-1 routed experts and shared experts
- checking sequence-level router load-balancing loss
- exercising custom tokenizer loading
- testing `generate()`, `save_pretrained()`, and `from_pretrained()`
- compact inference-engine and architecture experiments

It is not intended for:

- instruction following or chat
- factual question answering
- high-quality long-form generation
- production deployment
- safety-critical use
- benchmark comparison with production DeepSeek models

## Limitations

Known limitations include:

- only 2.93 million parameters
- small 1,003-token tokenizer
- English TinyStories-only pretraining
- weak factual knowledge and reasoning
- no instruction tuning or chat template
- occasional grammatical and semantic errors
- invented words and truncated sentences
- repetition and template-like stories
- no quality-equivalence claim with official DeepSeek models
- no current llama.cpp or GGUF inference support assumed for this exact
  DeepSeek V2 MLA/MoE configuration

## Notes on GGUF

The checkpoint is distributed as a normal float32 Hugging Face Safetensors
model. A useful GGUF build requires converter and runtime support for this
DeepSeek V2 MLA and MoE graph, including compressed KV projections, decoupled
RoPE, routed experts, and the shared expert. Merely placing tensors in a GGUF
container is not sufficient for compatible inference.

## Citation

This is a synthetic tiny DeepSeek V2-compatible MoE checkpoint trained from
scratch on TinyStories. It is intended for implementation validation,
debugging, education, and small-scale architecture experiments.