--- license: mit language: - en tags: - gpt_oss - moe - mixture-of-experts - causal-lm - tinystories - tiny-model - validation - debug-model - transformers pipeline_tag: text-generation --- # Tiny gpt-oss MoE 3M This repository contains a tiny gpt-oss-style text-only Mixture-of-Experts causal language model for validation and debugging. The model is intentionally small. It is not intended to be a high-quality text generation model. Its main purpose is to provide a compact checkpoint that exercises gpt-oss MoE text-model code paths in Hugging Face Transformers and in independent inference engines. This checkpoint is useful for implementation testing because it includes sliding attention and full attention layers, grouped-query attention, YaRN-style RoPE configuration, untied input/output embeddings, and top-4 MoE routing with multiple local experts. This is a synthetic tiny validation checkpoint. It is not an official OpenAI model and does not contain weights from the original gpt-oss checkpoints. ## Model purpose This model is designed for: * testing `GptOssForCausalLM` * validating `GptOssConfig` * testing gpt-oss-style MoE model loading * checking model save/load behavior * checking tokenizer save/load behavior * exercising sliding attention layers * exercising full attention layers * exercising grouped-query attention * exercising YaRN-style RoPE configuration parsing * exercising untied input/output embedding paths * exercising MoE expert parameters * exercising top-4 expert routing * providing a compact gpt-oss-style MoE checkpoint for inference-engine validation It is not designed for: * high-quality story generation * instruction following * chat use * benchmark comparison against production language models * production deployment * reproducing the behavior of the original gpt-oss models * testing MXFP4 quantized weight loading ## Model architecture The model uses `GptOssForCausalLM` with a small gpt-oss-style MoE configuration. Representative configuration: ```text model_type: gpt_oss vocab_size: 1024 hidden_size: 128 intermediate_size: 128 num_hidden_layers: 6 num_attention_heads: 4 num_key_value_heads: 1 head_dim: 32 sliding_window: 128 max_position_embeddings: 4096 initial_context_length: 1024 layer_types: - sliding_attention - full_attention - sliding_attention - full_attention - sliding_attention - full_attention hidden_act: silu tie_word_embeddings: false attention_bias: true attention_dropout: 0.0 rms_norm_eps: 1e-05 initializer_range: 0.02 rope_theta: 150000.0 rope_scaling: rope_type: yarn factor: 4.0 original_max_position_embeddings: 1024 beta_fast: 32.0 beta_slow: 1.0 truncate: false num_local_experts: 8 num_experts_per_tok: 4 experts_per_token: 4 output_router_logits: false router_aux_loss_coef: 0.0 swiglu_limit: 7.0 pad_token_id: 1000 bos_token_id: 1000 eos_token_id: 1001 ``` The attention pattern is: ```text sFsFsF ``` where `s` means `sliding_attention` and `F` means `full_attention`. This pattern was chosen for validation coverage. A full-attention-only model may be easier to train, but it would not exercise the sliding attention path. ## MoE configuration This model enables gpt-oss-style MoE blocks. ```text num_local_experts: 8 num_experts_per_tok: 4 experts_per_token: 4 intermediate_size: 128 ``` The `num_local_experts=8` and `num_experts_per_tok=4` setting is intentional. It keeps the model small while still exercising top-4 routing without selecting all experts on every token. This checkpoint is intended to cover: ```text router parameters multiple local experts top-4 expert selection weighted expert combination MoE FFN parameters sliding and full attention interaction GQA with num_key_value_heads = 1 untied input/output embeddings ``` ## Training data The model was trained on TinyStories-style English story text. The model uses a small custom byte-level BPE tokenizer. The tokenizer is designed for tiny validation models rather than production text quality. The configuration keeps the checkpoint compact and makes the model easier to train at very small scale. ## Tokenizer The reference training script uses a legacy byte-level BPE tokenizer setup: ```text RawTokenizer(BPE()) ByteLevel(add_prefix_space=False) BpeTrainer(vocab_size=1000, min_frequency=2, special_tokens=[], initial_alphabet=ByteLevel.alphabet()) normalizer: None ``` Special tokens are added after BPE training: ```text -> expected id 1000 -> expected id 1001 <|im_start|> -> expected id 1002 ``` The tokenizer uses: ```text bos_token: eos_token: pad_token: ``` The model config uses `vocab_size=1024` to leave a small reserved range above the learned base vocabulary and special tokens. This tokenizer setup was chosen because it produced substantially better tiny-model training behavior than the earlier tokenizer configuration where special tokens were included directly in the BPE training step. ## Training setup Representative training settings: ```text num_epochs: 1 learning_rate: 2e-4 batch_size: 32 block_size: 256 device: auto dtype: auto, resolved to float32 by the training script grad_clip: 1.0 error_on_nonfinite_gradients: true vocab_size: 1024 base_vocab_size: 1000 hidden_size: 128 intermediate_size: 128 num_hidden_layers: 6 num_attention_heads: 4 num_key_value_heads: 1 head_dim: 32 layer_pattern: sFsFsF sliding_window: 128 max_position_embeddings: 4096 initial_context_length: 1024 num_local_experts: 8 num_experts_per_tok: 4 router_aux_loss_coef: 0.0 attention_bias: true tie_word_embeddings: false ``` The final evaluation loss in the reference run was approximately: ```text Final loss: 1.4606 ``` This loss should not be interpreted as a general language-model quality benchmark. The model is very small and includes gpt-oss-specific architectural paths primarily for validation coverage. ## Example generation Example output from the reference checkpoint: ```text Prompt: Once upon Once upon a time, there was a little girl named Lily. She loved to play outside in the sun. One day, she saw a big, scary dog. The dog was barking and running around. Lily wanted to pet the dog, but she didn't want to leave. Suddenly, a big dog came running towards her. Lily was scared and didn't know what to do. But then she remembered that she had a friend ``` ```text Prompt: There was a little There was a little girl named Lily who was very curious. She wanted to know what was inside the box. She asked her mom, "What is in the box?" Her mom said, "It's a box of candy. It's a special treat for you." Lily was excited to open the box. She opened the box and found a big, shiny candy inside. She was so happy and said, "Thank you, Mommy!" Her mom smiled and ``` ```text Prompt: One day One day, a little girl named Lily went to the park with her mom. She saw a big slide and wanted to try it. She asked her mom if she could go on the slide. Her mom said yes, but only if she promised to be careful. Lily was so happy that she ran up the slide and started to slide down. She slid down the slide and laughed as she slid down. She felt so happy and free. ``` The model can generate TinyStories-like text fragments, but repetition, template collapse, and weak long-form coherence are expected. This is normal for this checkpoint and is not considered a failure for its intended purpose. ## Usage ```python import torch from transformers import PreTrainedTokenizerFast, GptOssForCausalLM repo = "shibatch/tinygptossmoe3m" tokenizer = PreTrainedTokenizerFast.from_pretrained(repo, subfolder="hf") model = GptOssForCausalLM.from_pretrained( repo, subfolder="hf", torch_dtype=torch.float32, ) model.eval() prompt = "Once upon" inputs = tokenizer(prompt, return_tensors="pt") with torch.no_grad(): output_ids = model.generate( **inputs, max_new_tokens=100, do_sample=False, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id, ) print(tokenizer.decode(output_ids[0], skip_special_tokens=True)) ``` ## Loading requirements This checkpoint requires a Transformers version that supports gpt-oss models. The following imports should work: ```python from transformers import GptOssForCausalLM, GptOssConfig ``` If these imports fail, update Transformers to a version with gpt-oss support. ## Tokenizer loading note This repository uses a custom byte-level BPE tokenizer saved as a `PreTrainedTokenizerFast`. For this reason, examples use: ```python from transformers import PreTrainedTokenizerFast ``` instead of `AutoTokenizer`. Using `AutoTokenizer` may fail in some environments if the tokenizer backend cannot be inferred automatically. The expected tokenizer files include: ```text tokenizer.json tokenizer_config.json special_tokens_map.json ``` ## Intended validation coverage This checkpoint is intended to validate support for: ```text GptOssConfig GptOssForCausalLM sliding_attention layers full_attention layers GQA with num_key_value_heads = 1 YaRN-style rope_scaling config parsing attention_bias = true untied input/output embeddings gpt-oss RMSNorm behavior gpt-oss MLP activation path gpt-oss MoE expert parameters num_local_experts = 8 num_experts_per_tok = 4 experts_per_token = 4 MoE expert dispatch MoE expert output combination generate() save_pretrained() from_pretrained() ``` ## Limitations This is a tiny debug model. It should not be used as a general-purpose language model. Known limitations: * frequent phrase repetition * TinyStories template collapse * weak long-form coherence * small vocabulary * weak semantic consistency * no instruction tuning * no chat formatting * no production use * no MXFP4 checkpoint coverage * no compatibility claim with original gpt-oss model quality The checkpoint is primarily intended to make gpt-oss-style MoE text-model code paths easy to test without downloading a large model. ## Why include MoE? A dense tiny model is simpler to train, but it does not cover MoE-specific implementation paths. This checkpoint intentionally includes: ```text num_local_experts = 8 num_experts_per_tok = 4 ``` to exercise a more realistic top-4 MoE routing path than a minimal `top_k=1` configuration. ## Why not full attention only? A full-attention-only tiny model may train more cleanly, but it would not cover gpt-oss-style sliding attention behavior. This checkpoint uses: ```text sliding_attention full_attention sliding_attention full_attention sliding_attention full_attention ``` to cover both attention implementations. ## Notes on MXFP4 and quantization The original large gpt-oss checkpoints use specialized quantization settings. This tiny validation checkpoint is trained and saved as a normal small floating-point Hugging Face model. It is intended for architecture, loader, routing, and generation validation. It is not intended to validate MXFP4 decoding or production quantized loading paths. A quantized tiny gpt-oss validation checkpoint can be produced as a separate artifact if needed. ## Suggested repository name Suggested Hugging Face repository name: ```text shibatch/tinygptossmoe3m ``` ## Citation This is a synthetic tiny validation checkpoint derived from gpt-oss-compatible MoE text architecture settings. It is intended for debugging and implementation testing.