license: isc
language:
- en
library_name: transformers
pipeline_tag: text-generation
datasets:
- roneneldan/TinyStories
tags:
- gemma4
- text-generation
- tiny-llm
- tinystories
- experimental
model-index:
- name: tinygemma4
results:
- task:
type: text-generation
name: Text Generation
dataset:
type: roneneldan/TinyStories
name: TinyStories validation
metrics:
- type: loss
name: validation loss
value: 2.2904
- type: perplexity
name: validation perplexity
value: 9.88
tinygemma4
tinygemma4 is a deliberately tiny, text-only Gemma 4 architecture experiment trained from scratch on TinyStories. It is intended for architecture compatibility checks, inference-engine testing, and small-scale language-model experiments.
This is not a useful assistant model. It was trained on simple synthetic stories and should be expected to produce short, child-story-like completions with limited coherence.
Model Details
- Architecture:
Gemma4TextForCausalLM - Parameters: 4,964,764
- Vocabulary: 8192-token byte-level BPE
- Context length in config: 2048
- Training block size: 256
- Hidden size: 128
- Per-layer input hidden size: 16
- Layers: 12
- Attention heads: 4
- KV heads: 1
- Head dimension: 32
- MLP intermediate size: 384
- Sliding window: 128
- Full attention layers: 4, 8, 12
- Embeddings: tied
- MoE: disabled
- Multimodal components: none
- Tensor format: safetensors
The checkpoint is saved in ordinary Hugging Face Transformers format. Any runtime with a correct Gemma 4 text implementation and support for these small dimensions should be able to load it.
Training
- Dataset:
roneneldan/TinyStories - Training file:
TinyStoriesV2-GPT4-train.txt - Validation file:
TinyStoriesV2-GPT4-valid.txt - Final training step: 300000
- Optimizer: AdamW
- Hardware: AMD Radeon RX 9070 XT, ROCm PyTorch for Windows
- Training dtype: bf16 autocast where available
Evaluation
Validation was measured during training on held-out TinyStories text with the local training script:
- Validation loss: 2.2904
- Validation perplexity: 9.88
These numbers are only for this training setup. They are not general language-understanding benchmarks.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "ApexDevelopment/tinygemma4"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
inputs = tokenizer("Once upon a time,", return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=80,
do_sample=True,
temperature=0.8,
top_p=0.95,
pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Limitations
- The model is tiny and heavily capacity-limited.
- It is trained only on synthetic TinyStories text.
- It is not instruction tuned.
- It is not safety tuned.
- It can repeat, contradict itself, or produce malformed story fragments.
- It should be used for experimentation and testing, not production.
Data and License Notes
The training dataset card lists TinyStories under cdla-sharing-1.0. This model was trained from scratch; it does not contain Gemma weights from Google or weights from TinyLLama-v0.
Weights are released under the license declared in the metadata above. Users are responsible for checking whether their intended use is compatible with the dataset license and applicable law.
Inspiration
This project was inspired by Maykeye/TinyLLama-v0, but uses a Gemma 4 text configuration instead of Llama.