Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,14 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tinymoeja2m.BF16.gguf filter=lfs diff=lfs merge=lfs -text
+tinymoeja2m.F16.gguf filter=lfs diff=lfs merge=lfs -text
+tinymoeja2m.F32.gguf filter=lfs diff=lfs merge=lfs -text
+tinymoeja2m.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+tinymoeja2m.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+tinymoeja2m.Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+tinymoeja2m.Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
+tinymoeja2m.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+tinymoeja2m.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+tinymoeja2m.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+tinymoeja2m.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,149 @@
+---
+license: mit
+base_model: mistralai/Mixtral-8x7B-v0.1
+tags:
+- mixtral
+- moe
+- gguf
+- safetensors
+- transformers
+- validation
+- test-suite
+- japanese
+---
+
+# TinyStories Mixtral 2M Top-2 MoE GQA Japanese Validation Suite (tinymoeja2m)
+
+This repository provides an ultra-lightweight, Japanese-specialized Mixtral model variant scaled down to a **2.05M total parameter footprint** and a **1.14M active parameter execution frame**. It is trained on the comprehensive 320k Japanese translated stories from the TinyStories dataset via Gemma 4.
+
+This asset is configured with a **2,048 token context window (2k)** and a standard RoPE base frequency (`rope_theta`) of **10,000.0** to act as a clean, trick-free baseline validation asset for runtime implementations.
+
+It is designed specifically for debugging custom inference engines against the synergy of Grouped-Query Attention (GQA) and Mixture-of-Experts (MoE) topologies.
+
+---
+
+## 📊 Comparison: `tinymoeja2m` vs Other Variants
+
+To help track feature coverage across the verification suite, the updated structural layouts are outlined below:
+
+| Feature / Metric | `tiny1m` (Standard) | `tinygemma1m` (Gemma 2) | `tinymoe2m` (English 4k) | `tinymoeja2m` (This Repository) |
+| :--- | :--- | :--- | :--- | :--- |
+| **Language** | English | English | English | **Japanese** |
+| **Base Architecture** | Llama 2 | Gemma 2 | Llama 2 (Mixtral) | **Llama 2 (Mixtral Format)** |
+| **FFN Structure** | Single FFN (Dense) | Single FFN (Dense) | Mixture-of-Experts | **Mixture-of-Experts (MoE)** |
+| **Attention Mechanism** | MHA (Multi-Head) | GQA (Grouped-Query) | MHA (Multi-Head) | **GQA (Grouped-Query)** |
+| **Total / Selected Experts**| 1 / - | 1 / - | 4 Experts / Top-2 | **4 Experts / Top-2** |
+| **GQA Head Ratio (Q:KV)** | 1:1 (MHA) | 4:1 (GQA) | 1:1 (MHA) | **4:1 (Query: 4, KV: 1)** |
+| **Max Position Embeddings** | - | - | 4,096 | **2,048 (2k Context)** |
+| **RoPE Base (`rope_theta`)** | - | - | 15,000.0 | **10,000.0** |
+| **Total / Active Params** | ~1.2M / ~1.2M | ~1.0M / ~1.0M | ~1.95M / ~1.14M | **~2.05M / ~1.14M** |
+| **Primary Debug Target** | Core matrix mult | Advanced graph | Scatter/Gather loops | **GQA Broadcast & Byte Fallback** |
+
+---
+
+## 📂 Repository Structure & File Descriptions
+
+### 1. GGUF Formats (Root Directory `./`)
+Binary files optimized for execution via `llama.cpp` or compatible lower-level inference engines. Upstream parsers automatically recognize this under the `mixed` (Mixtral) descriptor.
+
+| Filename | Type | Target / Validation Focus |
+| :--- | :--- | :--- |
+| **`tinymoeja2m.F32.gguf`** | `F32` | **Baseline Test.** Eliminates quantization noise to isolate and verify raw probability mathematics. |
+| **`tinymoeja2m.F16.gguf`**<br>**`tinymoeja2m.BF16.gguf`** | `F16`<br>`BF16` | **Half-Precision Test.** Evaluates 16-bit floating-point unpacking routines and parallelized accumulation layers. |
+| **`tinymoeja2m.Q8_0.gguf`** | `Q8_0` | **Standard Quantization.** Verifies block-based uniform scaling across decentralized MoE structures. |
+| **`tinymoeja2m.Q4_0.gguf`**<br>**`tinymoeja2m.Q4_1.gguf`** | `Q4_0`<br>`Q4_1` | **Classic 4-bit Quantization.** Tests basic linear scaling and unpacking logic across multiple discontinuous expert weight matrices. |
+| **`tinymoeja2m.Q2_K.gguf`** | `Q2_K` | **Standard K-Quant (2-bit).** Evaluates mixed super-block dequantization loops feeding sparse FFN routines. |
+| **`tinymoeja2m.Q3_K_M.gguf`** | `Q3_K_M` | **Standard K-Quant (3-bit).** Tests sub-variant multi-block layouts handling dynamic routing vectors. |
+| **`tinymoeja2m.Q4_K_M.gguf`** | `Q4_K_M` | **Standard K-Quant (4-bit).** Target for modern 4-bit super-block logic coupled with sparse MoE graphs. |
+| **`tinymoeja2m.Q5_K_M.gguf`** | `Q5_K_M` | **Standard K-Quant (5-bit).** Validates high-fidelity mixed 5-bit precision layouts. |
+| **`tinymoeja2m.Q6_K.gguf`** | `Q6_K` | **Standard K-Quant (6-bit).** Validates 6-bit high-fidelity super-block dequantization. |
+
+### 2. Hugging Face Native Format (`./hf/`)
+Unquantized components formatted for direct instantiation inside the PyTorch `transformers` library ecosystem:
+* **`hf/model.safetensors`**: Raw unquantized matrix parameters containing all 4 expert sub-networks, GQA projection matrices, and the master router tensor.
+* **`hf/config.json`**: Architectural specifications built around `MixtralConfig`. Fully configured to enforce `num_attention_heads: 4`, `num_key_value_heads: 1`, `max_position_embeddings: 2048`, and `rope_theta: 10000.0`.
+* **`hf/generation_config.json`**: Standard generation defaults.
+* **`hf/tokenizer.model`**: The custom 1,024-vocabulary size SentencePiece BPE master binary trained on a clean Japanese text subset with `byte_fallback` enabled.
+* **`hf/tokenizer.json`**: Evaluated JSON-serialized token maps for high-speed interoperability across native tokenization backends.
+* **`hf/tokenizer_config.json`**: Enforced metadata linking `LlamaTokenizer` classes to guarantee correct handling of prefix spacing and automatic `<s>` (BOS) injection.
+* **`hf/special_tokens_map.json`**: Structural map linking special tokens (`<s>`=1, `</s>`=2, `<unk>`=0, `<pad>`=2).
+
+---
+
+## 🎯 Purpose & Design Philosophy (Verification Targets)
+
+This checkpoint is specifically engineered as a deterministic validation test asset for runtime computing backends and **is not designed for practical semantic tasks.**
+
+Due to the compact parameter size (~2.05M) and ultra-focused vocabulary layout (1,024 tokens), the network concentrates its capacity entirely on mastering Japanese phrase continuations and basic syntax under an autoregressive framework. 
+
+### Critical Debugging Capabilities for Custom Engines:
+1. **GQA Broadcast Matrix Multiplication**
+   The 4:1 Grouped-Query Attention structure requires the execution kernels to correctly share a single Key/Value cache block across 4 independent Query heads. This serves as an ideal testbed for tracking memory stride offsets and tensor broadcasting alignment in parallel computing shaders.
+2. **Multi-Byte UTF-8 Byte Fallback Validation**
+   With the vocabulary limited to 1,024 tokens, any kanji or character outside the primary training subset triggers the `byte_fallback` mechanism, breaking the character down into raw sequential UTF-8 byte tokens (3 tokens per character). This enforces a rigorous stress test on the engine's streaming decoder to correctly stitch unaligned byte streams back into flawless Japanese text without truncation or corruption.
+
+---
+
+## 🚀 Usage Examples
+
+### A. Running GGUF via llama.cpp
+To process the GQA MoE execution graph and evaluate dynamic expert routing directly on your shell:
+```bash
+./llama-cli -m tinymoeja2m.Q4_K_M.gguf -p "トムとリリーは" -n 64 --temp 0.0
+
+```
+
+### B. Loading Hugging Face Formats via Python
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+repo_id = "shibatch/tinymoeja2m"
+
+print("Loading Japanese MoE GQA configuration and tokenizer layers...")
+tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder="hf")
+model = AutoModelForCausalLM.from_pretrained(repo_id, subfolder="hf")
+
+device = "cuda" if torch.cuda.is_available() else ("xpu" if torch.xpu.is_available() else "cpu")
+model = model.to(device)
+model.eval()
+
+prompt = "トムとリリーは"
+inputs = tokenizer(prompt, return_tensors="pt").to(device)
+
+print("Running inference loop (Validating 4:1 GQA & Top-2 MoE Kernels)...")
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs, 
+        max_length=64, 
+        do_sample=False
+    )
+    
+generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+print("\n--- Inference Test Result ---")
+print("Prompt   :", prompt)
+print("Generated:", generated_text)
+
+```
+
+---
+
+## 📝 Model Specifications
+
+* **Architecture:** Mixtral (`MixtralForCausalLM`)
+* **Dataset:** TinyStories Japanese Translation Corpus (320k stories)
+* **Total Parameters (`num_local_experts` = 4):** ~2.05M
+* **Active Parameters (`num_experts_per_tok` = 2):** ~1.14M
+* **Vocabulary Size (`vocab_size`):** 1,024 (Custom SentencePiece BPE with `byte_fallback` enabled)
+* **Hidden Size (`hidden_size`):** 128
+* **Number of Hidden Layers (`num_hidden_layers`):** 3
+* **Number of Attention Heads (`num_heads` / `num_kv_heads`):** 4 / 1 *(4:1 GQA layout)*
+* **Individual Expert Internal Dimension (`intermediate_size`):** 352 *(SwiGLU structure)*
+* **Max Position Embeddings (`max_position_embeddings`):** 2,048
+* **RoPE Base Frequency (`rope_theta`):** 10,000.0
+
+## 📜 License
+
+* **License:** **MIT License**. You are completely free to duplicate, modify, distribute, and utilize these assets across any commercial, personal, or educational environments.

hf/config.json

@@ -0,0 +1,35 @@
+{
+  "architectures": [
+    "MixtralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_dim": null,
+  "hidden_act": "silu",
+  "hidden_size": 128,
+  "initializer_range": 0.02,
+  "intermediate_size": 352,
+  "max_position_embeddings": 2048,
+  "model_type": "mixtral",
+  "num_attention_heads": 4,
+  "num_experts_per_tok": 2,
+  "num_hidden_layers": 3,
+  "num_key_value_heads": 1,
+  "num_local_experts": 4,
+  "output_router_logits": false,
+  "pad_token_id": 2,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "router_aux_loss_coef": 0.001,
+  "router_jitter_noise": 0.0,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.9.0",
+  "use_cache": false,
+  "vocab_size": 1024
+}

hf/generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 2,
+  "transformers_version": "5.9.0",
+  "use_cache": true
+}

hf/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e05efbaa834a2cf2471dba0dde6bc25b5170583fcf51e212a6821397db542fe0
+size 8044808

hf/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "pad_token": "</s>",
+  "unk_token": "<unk>"
+}

hf/tokenizer.json

@@ -0,0 +1,107 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<unk>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "</s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "Metaspace",
+    "replacement": "▁",
+    "prepend_scheme": "first",
+    "split": false
+  },
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {}
+  },
+  "decoder": {
+    "type": "Sequence",
+    "decoders": [
+      {
+        "type": "Replace",
+        "pattern": {
+          "String": "▁"
+        },
+        "content": " "
+      },
+      {
+        "type": "ByteFallback"
+      },
+      {
+        "type": "Fuse"
+      },
+      {
+        "type": "Strip",
+        "content": " ",
+        "start": 1,
+        "stop": 0
+      }
+    ]
+  },
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": null,
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": true,
+    "byte_fallback": true,
+    "ignore_merges": false,
+    "vocab": {
+      "<unk>": 0,
+      "<s>": 1,
+      "</s>": 2
+    },
+    "merges": []
+  }
+}

hf/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:064eec0d246d749175b47e112fd8e16a69745af947b58a8bbce1fc93a4a1d0a7
+size 254715

hf/tokenizer_config.json

@@ -0,0 +1,12 @@
+{
+  "add_prefix_space": null,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

tinymoeja2m.BF16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfebfcdf0713b7f3abf564be699254d9eb1976f816216ecf6e5ad986e10a7905
+size 4048288

tinymoeja2m.F16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02ab7e1f621602db54b2f254fee0482b713c7ef25371e16037ab2c09b619e933
+size 4048288

tinymoeja2m.F32.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc6361d000efb38ff380f7b0cbe011e7f7714cf16edf71f69a786f39acee8abd
+size 8062368

tinymoeja2m.Q2_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ebf693e9b852052e06d330b00d4362999d8fac25faf793bb65dc988b6f481d1
+size 1230240

tinymoeja2m.Q3_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:446a7bc7adc26448ca4f5cde8d1e250906fcc04316b260b50d6301fffe949877
+size 1304480

tinymoeja2m.Q4_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe7997b4d6fe1034fcb6d1a61822cfc0a431958c0246341f3f45a2ed9eb7794f
+size 1228704

tinymoeja2m.Q4_1.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e3b2602948c914c0e399e47931de56b932fc62906eeee12ba5128b828525f66
+size 1345952

tinymoeja2m.Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd957f6eaa68b8f87546afbcb25f0af17fccc8d54ab074f7aa707bc5b6f35452
+size 1532320

tinymoeja2m.Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26400fec736c23f0728dfaaf3ae8ad61b4b2b55c666af4eff0c166999392b48c
+size 1638048

tinymoeja2m.Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3972decba22f8318a13a94ed58fa0ae077e0681d237dd7c8725f482734c73b6
+size 2166688

tinymoeja2m.Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e300cd576f0f5571f7d8e6e0ad022d5985c849932c5208a14478b1a0ad05e0a2
+size 2166688