Upload folder using huggingface_hub

.gitattributes

@@ -5,6 +5,7 @@
 *.ckpt filter=lfs diff=lfs merge=lfs -text
 *.ftz filter=lfs diff=lfs merge=lfs -text
 *.gz filter=lfs diff=lfs merge=lfs -text
+*.gguf filter=lfs diff=lfs merge=lfs -text
 *.h5 filter=lfs diff=lfs merge=lfs -text
 *.joblib filter=lfs diff=lfs merge=lfs -text
 *.lfs.* filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Naoki Shibata
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,108 +1,210 @@
 ---
+library_name: transformers
+pipeline_tag: text-generation
+language:
+- en
 license: mit
+datasets:
+- roneneldan/TinyStories
 tags:
 - gemma3
 - safetensors
 - transformers
+- gguf
+- q4-k-m
 - tinygemma
 - tinystories
 - validation
 - test-suite
 ---
 
-# TinyStories Gemma3 Text Validation Artifact
+# TinyStories Gemma3 2M (HF + Q4_K_M GGUF)
+
+This repository contains a tiny Gemma 3 text-only model trained with official
+Hugging Face Transformers classes, together with a llama.cpp-compatible
+Q4_K_M GGUF conversion.
+
+The model has 1,560,064 parameters (the `2m` name is an approximate size
+label). It is intended for inference-engine validation and small-model
+experiments, not for production language quality.
+
+The Hugging Face artifact was downloaded from
+[`shibatch/tinygemma3-2m`](https://huggingface.co/shibatch/tinygemma3-2m) at
+revision `7aab2f4e525707e046799eb5674f344dc74853d6`.
+
+## Repository contents
+
+- `hf/`: original Hugging Face model and tokenizer
+- `gguf/tinygemma3-2m-Q4_K_M.gguf`: ready-to-run Q4_K_M GGUF
+- `convert_to_gguf.py`: reproducible HF -> F16 -> Q4_K_M conversion wrapper
+- `conversion_metadata.json`: source revisions, quantization details, and
+  validation result
+- `reference/`: original deterministic Hugging Face reference tensors
+- `gemma3_text_config_dump.json`: normalized configuration dump
+- `safetensors_keys.json`: source tensor names and shapes
+- `artifact_metadata.json`: original training metadata
+- `SHA256SUMS`: checksums for the distributed binary artifacts
+
+## GGUF quick start
+
+Install a recent llama.cpp build that supports Gemma 3. On Debian systems with
+the llama.cpp tools package installed, run:
+
+```bash
+llama-completion \
+  -m gguf/tinygemma3-2m-Q4_K_M.gguf \
+  -p "Once upon" \
+  -n 100 \
+  --temp 0.8 \
+  --top-p 0.95 \
+  --seed 1234
+```
 
-This directory contains a tiny Gemma 3 text-only model trained with official
-Hugging Face Transformers classes.
+The GGUF was load- and generation-tested with Debian llama.cpp build 8681.
+For a deterministic smoke test:
+
+```bash
+llama-completion \
+  -m gguf/tinygemma3-2m-Q4_K_M.gguf \
+  -p "Once upon" \
+  -n 40 \
+  --temp 0 \
+  --seed 1234 \
+  --no-display-prompt
+```
 
-It is intended for inference-engine validation, not for production language
-quality.
+One validated completion was:
 
-## Official classes used
+```text
+ a time, there was a little girl named Lily. She loved to play with her toys
+ and her favorite thing to do was to go to the park. One day, Lily's
+```
 
-- `Gemma3TextConfig`
-- `Gemma3ForCausalLM`
-- `Trainer`
+## Q4_K_M details
 
-No custom Gemma 3 modeling code is used.
+The distributed GGUF reports `general.file_type = 15` (Q4_K_M) and contains
+80 tensors. Because this deliberately tiny architecture has a hidden width of
+128 while K-quants use 256-value blocks, llama.cpp applies its normal Q4_K_M
+fallback rules to tensors that cannot use Q4_K directly:
 
-## Key validation targets
+| Tensor type | Count |
+| --- | ---: |
+| F32 | 37 |
+| Q5_0 | 34 |
+| Q4_K | 4 |
+| Q8_0 | 3 |
+| Q6_K | 2 |
 
-- `model_type = gemma3_text`
-- `architectures = Gemma3ForCausalLM`
-- local/global attention pattern through `layer_types`
-- sliding-window attention
-- full attention
-- GQA
-- per-head `q_norm` / `k_norm`
-- Gemma3 four-norm decoder layer structure
-- gated MLP: `silu(gate_proj(x)) * up_proj(x)`
-- tied output head through `model.embed_tokens.weight`
+This mixed tensor layout is the expected llama.cpp representation of the
+Q4_K_M preset for this model shape. The file is about 1.1 MB, compared with
+about 3.1 MB for the intermediate F16 GGUF.
 
-## Tiny architecture
+## Reproduce the GGUF
 
-- vocab_size: 1024
-- hidden_size: 128
-- intermediate_size: 512
-- num_hidden_layers: 6
-- num_attention_heads: 4
-- num_key_value_heads: 1
-- head_dim: 32
-- sliding_window: 32
-- layer_types: ['sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'full_attention']
+Requirements:
 
-## Files
+- a recent llama.cpp source checkout containing `convert_hf_to_gguf.py`
+- `llama-quantize` on `PATH`, or its path passed explicitly
+- the Python packages required by llama.cpp's converter
 
-- `hf/`: Hugging Face model/tokenizer artifact
-- `reference/reference.pt`: deterministic reference tensors
-- `reference/reference.json`: JSON summary of reference logits
-- `gemma3_text_config_dump.json`: normalized config dump
-- `safetensors_keys.json`: tensor names and shapes
-- `artifact_metadata.json`: generation metadata
+Run from this repository:
 
-## Usage
+```bash
+python convert_to_gguf.py \
+  --llama-cpp /path/to/llama.cpp \
+  --quantizer /path/to/llama-quantize
+```
+
+The wrapper handles two properties of this validation checkpoint that generic
+conversion currently does not infer correctly:
+
+1. The custom tokenizer is a GPT-2-style ByteLevel BPE whose probe hash is not
+   in llama.cpp's generated pre-tokenizer table.
+2. The tokenizer has 1,003 entries, but the model intentionally pads its
+   embedding and logits vocabulary to 1,024 rows. Those rows must be retained
+   for Gemma 3 runtime shape checks and for compatibility with the reference
+   logits.
+
+The script validates the tokenizer structure before applying the tokenizer
+override. It creates the F16 intermediate inside `gguf/` and removes it after
+Q4_K_M quantization, including when quantization fails.
+
+## Hugging Face usage
 
 ```python
+from pathlib import Path
+
 import torch
 from transformers import Gemma3ForCausalLM, PreTrainedTokenizerFast
 
-def main():
-    repo_id = "shibatch/tinygemma3-2m"
-
-    print("Loading tokenizer...")
-    tokenizer = PreTrainedTokenizerFast.from_pretrained(repo_id, subfolder="hf")
-
-    print("Loading Gemma3 model weights...")
-    device = "cuda" if torch.cuda.is_available() else "cpu"
-
-    model = Gemma3ForCausalLM.from_pretrained(
-        repo_id,
-        subfolder="hf",
-        torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
-    ).to(device)
-    model.eval()
-
-    prompt = "Once upon"
-    print(f"\nInput prompt: {prompt}")
-
-    input_ids = tokenizer.encode(prompt, add_special_tokens=False)
-    input_ids = [tokenizer.bos_token_id] + input_ids
-    input_ids = torch.tensor([input_ids], dtype=torch.long, device=device)
-
-    with torch.no_grad():
-        outputs = model.generate(
-            input_ids,
-            max_new_tokens=100,
-            do_sample=False,
-            repetition_penalty=1.0,
-            top_p=1.0,
-            pad_token_id=tokenizer.pad_token_id or tokenizer.bos_token_id,
-            eos_token_id=tokenizer.eos_token_id,
-        )
-
-    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
-    print(f"Generated output: {generated_text}")
-
-if __name__ == "__main__":
-    main()
+model_dir = Path("hf")
+tokenizer = PreTrainedTokenizerFast.from_pretrained(model_dir)
+model = Gemma3ForCausalLM.from_pretrained(
+    model_dir,
+    dtype=torch.float32,
+).eval()
+
+prompt = "Once upon"
+input_ids = torch.tensor(
+    [[tokenizer.bos_token_id] + tokenizer.encode(prompt, add_special_tokens=False)]
+)
+
+with torch.no_grad():
+    output = model.generate(
+        input_ids,
+        max_new_tokens=100,
+        do_sample=False,
+        pad_token_id=tokenizer.pad_token_id,
+        eos_token_id=tokenizer.eos_token_id,
+    )
+
+print(tokenizer.decode(output[0], skip_special_tokens=True))
 ```
+
+The same example is available as `example_generate.py`.
+
+## Official classes used
+
+- `Gemma3TextConfig`
+- `Gemma3ForCausalLM`
+- `Trainer`
+
+No custom Gemma 3 modeling code is used by the original HF artifact.
+
+## Architecture
+
+```yaml
+model_type: gemma3_text
+architecture: Gemma3ForCausalLM
+parameter_count: 1,560,064
+vocab_size: 1,024
+tokenizer_entries: 1,003
+hidden_size: 128
+intermediate_size: 512
+num_hidden_layers: 6
+num_attention_heads: 4
+num_key_value_heads: 1
+head_dim: 32
+max_position_embeddings: 256
+sliding_window: 32
+layer_types:
+  - sliding_attention
+  - sliding_attention
+  - sliding_attention
+  - sliding_attention
+  - sliding_attention
+  - full_attention
+tie_word_embeddings: true
+```
+
+The model exercises local/global attention, sliding-window attention, GQA,
+per-head `q_norm` / `k_norm`, Gemma 3's four-norm decoder structure, a gated
+MLP, and a tied output head.
+
+## Limitations
+
+This is a synthetic tiny checkpoint. It is not an official Google model and
+does not contain weights from an original Gemma checkpoint. It is not intended
+for instruction following, chat, factual recall, safety-critical use, or
+production deployment. Quantized output may differ from the float32 Hugging
+Face checkpoint.

SHA256SUMS

@@ -0,0 +1,4 @@
+54e9a48a7d571b01e291f897ac03953b3aae3ab76acfd4d7e3d314f7efc47644  gguf/tinygemma3-2m-Q4_K_M.gguf
+ed87076e1e52de8e315b1e9fe2a551385fcedaed6d11c9c1ac8264804914ac61  hf/model.safetensors
+f21db7a5f75e39b8db16b83937d01fe6ef3e8c888a06fcfdc0aaca8bb0e242e2  reference/reference.pt
+

conversion_metadata.json

@@ -0,0 +1,42 @@
+{
+  "source": {
+    "repo_id": "shibatch/tinygemma3-2m",
+    "revision": "7aab2f4e525707e046799eb5674f344dc74853d6",
+    "downloaded_utc_date": "2026-08-13"
+  },
+  "conversion": {
+    "script": "convert_to_gguf.py",
+    "llama_cpp_source_id": "0b1bad14ff204627636aeb1de22ddcd5acb859d4",
+    "intermediate_type": "F16",
+    "quantizer": "llama.cpp 8681 (Debian)",
+    "requested_quantization": "Q4_K_M",
+    "general_file_type": 15,
+    "tensor_count": 80,
+    "tensor_type_counts": {
+      "F32": 37,
+      "Q5_0": 34,
+      "Q4_K": 4,
+      "Q8_0": 3,
+      "Q6_K": 2
+    },
+    "fallback_quantized_tensor_count": 36,
+    "tokenizer_pre": "gpt-2",
+    "tokenizer_probe_sha256": "a7cd49e25128643b1b7df2df3a699e60225476307c755f246cacfd441d533a7b",
+    "padded_vocab_rows_preserved": 1024
+  },
+  "output": {
+    "path": "gguf/tinygemma3-2m-Q4_K_M.gguf",
+    "size_bytes": 1153312,
+    "sha256": "54e9a48a7d571b01e291f897ac03953b3aae3ab76acfd4d7e3d314f7efc47644"
+  },
+  "validation": {
+    "runtime": "llama.cpp 8681 (Debian)",
+    "load_succeeded": true,
+    "generation_exit_code": 0,
+    "prompt": "Once upon",
+    "max_new_tokens": 40,
+    "temperature": 0.0,
+    "completion": " a time, there was a little girl named Lily. She loved to play with her toys and her favorite thing to do was to go to the park. One day, Lily's"
+  }
+}
+

convert_to_gguf.py

@@ -0,0 +1,122 @@
+#!/usr/bin/env python3
+"""Convert this repository's Hugging Face checkpoint to Q4_K_M GGUF.
+
+The tokenizer is a small custom ByteLevel BPE.  Its behavior is GPT-2-style,
+but its tokenizer probe hash is not yet present in llama.cpp's generated
+pre-tokenizer lookup table.  This wrapper validates the tokenizer structure
+before supplying the corresponding ``gpt-2`` pre-tokenizer identifier.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import runpy
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--llama-cpp",
+        type=Path,
+        required=True,
+        help="Path to a llama.cpp checkout containing convert_hf_to_gguf.py",
+    )
+    quantizer = shutil.which("llama-quantize")
+    parser.add_argument(
+        "--quantizer",
+        type=Path,
+        default=Path(quantizer) if quantizer else None,
+        help="Path to llama-quantize (default: resolve it from PATH)",
+    )
+    parser.add_argument(
+        "--outfile",
+        type=Path,
+        default=Path("gguf/tinygemma3-2m-Q4_K_M.gguf"),
+    )
+    return parser.parse_args()
+
+
+def is_expected_bytelevel_bpe(model_dir: Path) -> bool:
+    tokenizer = json.loads((model_dir / "tokenizer.json").read_text())
+    return (
+        tokenizer.get("normalizer") is None
+        and tokenizer.get("model", {}).get("type") == "BPE"
+        and tokenizer.get("model", {}).get("byte_fallback") is False
+        and tokenizer.get("pre_tokenizer")
+        == {
+            "type": "ByteLevel",
+            "add_prefix_space": False,
+            "trim_offsets": False,
+            "use_regex": True,
+        }
+    )
+
+
+def main() -> None:
+    args = parse_args()
+    repo_dir = Path(__file__).resolve().parent
+    model_dir = repo_dir / "hf"
+    converter = args.llama_cpp.resolve() / "convert_hf_to_gguf.py"
+    quantizer = args.quantizer.resolve() if args.quantizer else None
+    outfile = args.outfile if args.outfile.is_absolute() else repo_dir / args.outfile
+    intermediate = outfile.parent / ".tinygemma3-2m-f16.intermediate.gguf"
+
+    if not converter.is_file():
+        raise SystemExit(f"llama.cpp converter not found: {converter}")
+    if quantizer is None or not quantizer.is_file():
+        raise SystemExit("llama-quantize not found; pass it with --quantizer")
+    if not is_expected_bytelevel_bpe(model_dir):
+        raise SystemExit("Unexpected tokenizer structure; refusing to guess GGUF metadata")
+
+    sys.path.insert(0, str(args.llama_cpp.resolve()))
+    from conversion import TextModel  # noqa: PLC0415
+    from conversion.gemma import Gemma3Model  # noqa: PLC0415
+
+    original = TextModel.get_vocab_base_pre
+
+    def get_vocab_base_pre(self: TextModel, tokenizer: object) -> str:
+        if Path(self.dir_model).resolve() == model_dir.resolve():
+            return "gpt-2"
+        return original(self, tokenizer)
+
+    def modify_tensors(
+        self: Gemma3Model, data_torch: object, name: str, bid: int | None
+    ) -> object:
+        # This checkpoint intentionally pads the embedding matrix and logits
+        # from 1,003 tokenizer entries to config.vocab_size=1,024.  Keep those
+        # rows so the GGUF architecture and HF reference logits stay aligned.
+        f_shift = self.norm_shift(name)
+        if f_shift != 0.0:
+            data_torch = data_torch + f_shift
+        yield from super(Gemma3Model, self).modify_tensors(data_torch, name, bid)
+
+    TextModel.get_vocab_base_pre = get_vocab_base_pre
+    Gemma3Model.modify_tensors = modify_tensors
+    outfile.parent.mkdir(parents=True, exist_ok=True)
+    sys.argv = [
+        str(converter),
+        str(model_dir),
+        "--outfile",
+        str(intermediate),
+        "--outtype",
+        "f16",
+        "--model-name",
+        "tinygemma3-2m",
+    ]
+    try:
+        runpy.run_path(str(converter), run_name="__main__")
+        subprocess.run(
+            [str(quantizer), str(intermediate), str(outfile), "Q4_K_M"],
+            check=True,
+        )
+    finally:
+        intermediate.unlink(missing_ok=True)
+
+
+if __name__ == "__main__":
+    main()

example_generate.py

@@ -0,0 +1,39 @@
+from pathlib import Path
+
+import torch
+from transformers import Gemma3ForCausalLM, PreTrainedTokenizerFast
+
+
+MODEL_DIR = Path(__file__).resolve().parent / "hf"
+PROMPT = "Once upon"
+
+
+def main() -> None:
+    tokenizer = PreTrainedTokenizerFast.from_pretrained(MODEL_DIR)
+    model = Gemma3ForCausalLM.from_pretrained(
+        MODEL_DIR,
+        dtype=torch.float32,
+    ).eval()
+
+    input_ids = torch.tensor(
+        [
+            [tokenizer.bos_token_id]
+            + tokenizer.encode(PROMPT, add_special_tokens=False)
+        ]
+    )
+
+    with torch.no_grad():
+        output = model.generate(
+            input_ids,
+            max_new_tokens=100,
+            do_sample=False,
+            pad_token_id=tokenizer.pad_token_id,
+            eos_token_id=tokenizer.eos_token_id,
+        )
+
+    print(tokenizer.decode(output[0], skip_special_tokens=True))
+
+
+if __name__ == "__main__":
+    main()
+

gguf/tinygemma3-2m-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:54e9a48a7d571b01e291f897ac03953b3aae3ab76acfd4d7e3d314f7efc47644
+size 1153312

requirements.txt

@@ -0,0 +1,4 @@
+torch>=2.6.0
+transformers>=5.9.0
+safetensors>=0.4.0
+