README.md

---
library_name: transformers
pipeline_tag: text-generation
language:
  - en
license: mit
datasets:
  - roneneldan/TinyStories
tags:
  - deepseek-v3
  - multi-head-latent-attention
  - mixture-of-experts
  - moe
  - tinystories
  - tiny-model
  - text-generation
  - validation
  - debug-model
---

# Tiny DeepSeek-V3 3M

This repository contains a tiny `DeepseekV3ForCausalLM` Mixture-of-Experts
language model trained from scratch on the full TinyStories training corpus.

The model has **2,803,272 total parameters** and approximately **1,807,944
parameters active per token**. It retains the main DeepSeek-V3 inference
building blocks at a deliberately small scale: Multi-Head Latent Attention
(MLA), Q and KV low-rank compression, interleaved RoPE, routed and shared
experts, sigmoid top-k routing, and correction-bias-based load balancing.

This is an independently trained synthetic tiny checkpoint. It is not an
official DeepSeek model, contains no weights from an original DeepSeek
checkpoint, and should not be expected to match the capabilities of production
DeepSeek models.

## Repository contents

- `hf/`: the final Hugging Face checkpoint and tokenizer
- `example_generate.py`: a minimal local generation example
- `eval_text_generation.json`: generations from the final training evaluation
- `artifact_metadata.json`: training arguments, metrics, router usage, and the
  expanded configuration
- `deepseek_v3_config_dump.json`: a standalone configuration dump

Optimizer checkpoints, packed training data, and the full training log are
intentionally omitted from the distribution package.

## Architecture identity

This checkpoint loads through the standard Transformers classes:

```text
DeepseekV3Config
DeepseekV3ForCausalLM
model_type: deepseek_v3
```

It does not use `DeepseekV2ForCausalLM` with a renamed model card. Important
V3-specific behavior includes sigmoid routing, normalized selected routing
weights, interleaved RoPE, Q LoRA compression, and an expert correction bias.

## Model architecture

```yaml
architecture: DeepseekV3ForCausalLM
model_type: deepseek_v3
total_parameter_count: 2,803,272
active_parameters_per_token: 1,807,944

model_vocab_size: 1,024
tokenizer_size: 1,003
hidden_size: 216
intermediate_size: 432
num_hidden_layers: 5
first_dense_layers: 1
moe_layers: 4

num_attention_heads: 8
num_key_value_heads: 8
q_lora_rank: 64
kv_lora_rank: 64
qk_nope_head_dim: 16
qk_rope_head_dim: 16
qk_head_dim: 32
v_head_dim: 32
rope_interleave: true
rope_theta: 10,000

routed_experts_per_moe_layer: 4
shared_experts_per_moe_layer: 1
experts_selected_per_token: 1
moe_intermediate_size: 128
n_group: 1
topk_group: 1
norm_topk_prob: true
routed_scaling_factor: 2.5

tie_word_embeddings: true
rms_norm_eps: 1.0e-6
max_position_embeddings: 2,048
```

### Multi-Head Latent Attention

Every layer uses DeepSeek MLA rather than conventional GQA. Queries are
compressed through a rank-64 Q projection, while keys and values share a
rank-64 latent representation before expansion to eight attention heads.
Each query/key head combines a 16-dimensional non-positional component and a
16-dimensional rotary component. Values use 32 dimensions per head.

`num_key_value_heads` equals `num_attention_heads` because MLA compresses the
shared KV latent state before expanding it to the attention heads. Setting a
smaller KV-head count would describe GQA, not this MLA implementation.

### DeepSeekMoE routing

The first decoder layer is dense. Each of the remaining four layers contains
four routed experts and one always-active shared expert. One routed expert is
selected for each token.

The V3 router computes sigmoid affinity scores. Selection uses the affinity
score plus a non-gradient correction bias, while the routed expert weight uses
the original affinity score. During training, the correction bias was adjusted
after every batch: overloaded experts were decreased and underloaded experts
were increased. A very small complementary sequence-wise balance objective was
also used.

```yaml
v3_bias_update_speed: 0.01
sequence_aux_loss_alpha: 0.0001
no_token_dropping: true
```

All routed experts received traffic. Aggregate fractions across the complete
training run ranged from approximately 0.140 to 0.496 depending on layer and
expert. These aggregate values reflect learned routing specialization and are
not expected to be exactly uniform.

## MTP scope

The checkpoint contains the standard Transformers DeepSeek-V3 causal language
model used for inference. It does **not** contain a training-only Multi-Token
Prediction (MTP) draft module:

```yaml
num_nextn_predict_layers: 0
training_objective: next-token prediction
```

Transformers 5.14.1 exposes the V3 MTP count in configuration but does not
instantiate or train the original DeepSeek-V3 MTP module in
`DeepseekV3ForCausalLM`. The checkpoint therefore preserves the V3 main-model
inference graph and V3 routing behavior, but does not claim to reproduce the
complete original V3 pretraining recipe.

## Training data

The model was trained on the full TinyStories training corpus using an
independent 1% validation split:

```yaml
selected_stories: 2,119,489
training_stories: 2,098,294
validation_stories: 21,195
validation_fraction: 0.01

training_blocks: 2,441,053
validation_blocks: 24,607
block_size: 256
```

Stories were joined into a continuous packed stream:

```text
BOS + story 1 + EOS + BOS + story 2 + EOS + ...
```

The stream was split into fixed 256-token blocks without per-story padding.
Only the final incomplete block was discarded.

## Tokenizer

The checkpoint uses a custom byte-level BPE tokenizer. The base vocabulary was
trained on the 5% TinyStories experiment and frozen for the full run:

```text
BPE()
ByteLevel(add_prefix_space=False)
base_vocab_size: 1,000
min_frequency: 2
normalizer: None
```

Special tokens use fixed IDs:

```text
<s>          -> 1000
</s>         -> 1001
<|im_start|> -> 1002
```

`<s>` is both BOS and padding, while `</s>` is EOS. The model reserves 1,024
embedding rows and the tokenizer exposes 1,003 tokens.

The presence of `<|im_start|>` does not make this a chat model. The checkpoint
is pretrained only and has no chat template or instruction tuning.

## Training setup

The checkpoint was trained from scratch in float32 on an NVIDIA GeForce RTX
5060 Ti:

```yaml
dtype: float32
batch_size: 16
block_size: 256
training_steps: 152,565
epochs: 1.0
tokens_processed: 624,906,240

optimizer: AdamW
learning_rate: 2.0e-4
warmup_steps: 2,000
scheduler: warmup + cosine decay
minimum_learning_rate: 2.0e-5
weight_decay: 0.0
grad_clip: 1.0

optimizer_loop_time: approximately 1h 03m 36s
```

## Evaluation

The final checkpoint produced:

```yaml
final_train_loss: 1.5201
validation_loss: 1.5254
validation_perplexity: 4.5972
```

Validation loss was computed over 32 batches, or 131,072 tokens, from the
independent packed validation split. These values are compact checkpoint
diagnostics, not general language-model benchmark results.

## Example generation

A representative final sampled generation begins:

```text
Once upon a time, there was a little girl named Lily. She had long black,
white hair that she loved to play with every day. One day, Lily's mom asked
her to clean up her bedroom. Lily didn't want to clean up, so she started to
pick up her toys and put them away in the closet.
```

An independent greedy reload test produced:

```text
Once upon a time, there was a little girl named Lily. She loved to play
outside in the sunshine and pick flowers. One day, she saw a big, scary dog
running towards her. The dog ran away and Lily was sad.
```

Sampling is stochastic. The model produces recognizable TinyStories-style
English, but contradictions, incorrect pronouns, invented words, repetition,
and unfinished stories remain possible at this size.

## Usage

Install the requirements:

```bash
pip install -r requirements.txt
```

Run the included example from the repository root:

```bash
python example_generate.py
```

Local loading:

```python
from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "shibatch/tinydeepseekv3-3m"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_dir = Path(snapshot_download(
    repo_id=repo,
    allow_patterns=["hf/*"],
)) / "hf"

tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    dtype=torch.float32,
).to(device).eval()

input_ids = torch.tensor(
    [[tokenizer.bos_token_id] + tokenizer.encode(
        "Once upon",
        add_special_tokens=False,
    )],
    dtype=torch.long,
    device=device,
)

with torch.inference_mode():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        top_k=40,
        repetition_penalty=1.1,
        use_cache=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

Loading the `hf` subdirectory from Hugging Face Hub:

```python
from pathlib import Path
from huggingface_hub import snapshot_download

repo_dir = Path(snapshot_download(
    repo_id="shibatch/tinydeepseekv3-3m",
    allow_patterns=["hf/*"],
))
model_dir = repo_dir / "hf"
```

Use `model_dir` with the local loading example above.

## Loading requirements

The model was trained, saved, and independently reloaded with:

```text
transformers 5.14.1
torch 2.12.0+cu130
```

It requires a Transformers release containing `DeepseekV3ForCausalLM`. No
custom model code or `trust_remote_code=True` is required.

## Intended uses

This model is intended for:

- testing `DeepseekV3Config` and `DeepseekV3ForCausalLM`
- testing Multi-Head Latent Attention and compressed KV caching
- testing Q and KV low-rank projections
- testing V3 sigmoid MoE routing and shared experts
- testing correction-bias-based expert load balancing
- compact inference-engine and architecture experiments
- testing `generate()`, `save_pretrained()`, and `from_pretrained()`

It is not intended for:

- instruction following or chat
- factual question answering
- reasoning benchmarks
- production deployment
- safety-critical use
- comparison with full-size DeepSeek models

## Limitations

- only 2.80 million total parameters
- small 1,003-token tokenizer
- English TinyStories-only pretraining
- no instruction tuning and no chat template
- no MTP training module or MTP objective
- weak factual knowledge and reasoning
- occasional grammatical and semantic errors
- possible mojibake inherited from training-text byte sequences
- no capability-equivalence claim with official DeepSeek models

## Citation and references

This is a synthetic tiny DeepSeek-V3-compatible MoE checkpoint trained from
scratch on TinyStories. It is intended for implementation validation,
debugging, education, and small-scale architecture experiments.

- [DeepSeek-V3 Technical Report](https://arxiv.org/abs/2412.19437)
- [Official DeepSeek-V3 repository](https://github.com/deepseek-ai/DeepSeek-V3)
- [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories)