---
license: isc
language:
  - en
library_name: transformers
pipeline_tag: text-generation
datasets:
  - roneneldan/TinyStories
tags:
  - gemma4
  - text-generation
  - tiny-llm
  - tinystories
  - experimental
model-index:
  - name: tinygemma4
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          type: roneneldan/TinyStories
          name: TinyStories validation
        metrics:
          - type: loss
            name: validation loss
            value: 2.2904
          - type: perplexity
            name: validation perplexity
            value: 9.88
---

# tinygemma4

tinygemma4 is a deliberately tiny, text-only Gemma 4 architecture experiment trained from scratch on TinyStories. It is intended for architecture compatibility checks, inference-engine testing, and small-scale language-model experiments.

This is not a useful assistant model. It was trained on simple synthetic stories and should be expected to produce short, child-story-like completions with limited coherence.

## Model Details

- Architecture: `Gemma4TextForCausalLM`
- Parameters: 4,964,764
- Vocabulary: 8192-token byte-level BPE
- Context length in config: 2048
- Training block size: 256
- Hidden size: 128
- Per-layer input hidden size: 16
- Layers: 12
- Attention heads: 4
- KV heads: 1
- Head dimension: 32
- MLP intermediate size: 384
- Sliding window: 128
- Full attention layers: 4, 8, 12
- Embeddings: tied
- MoE: disabled
- Multimodal components: none
- Tensor format: safetensors

The checkpoint is saved in ordinary Hugging Face Transformers format. Any runtime with a correct Gemma 4 text implementation and support for these small dimensions should be able to load it.

## Training

- Dataset: `roneneldan/TinyStories`
- Training file: `TinyStoriesV2-GPT4-train.txt`
- Validation file: `TinyStoriesV2-GPT4-valid.txt`
- Final training step: 300000
- Optimizer: AdamW
- Hardware: AMD Radeon RX 9070 XT, ROCm PyTorch for Windows
- Training dtype: bf16 autocast where available

## Evaluation

Validation was measured during training on held-out TinyStories text with the local training script:

- Validation loss: 2.2904
- Validation perplexity: 9.88

These numbers are only for this training setup. They are not general language-understanding benchmarks.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ApexDevelopment/tinygemma4"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")

inputs = tokenizer("Once upon a time,", return_tensors="pt")
outputs = model.generate(
	**inputs,
	max_new_tokens=80,
	do_sample=True,
	temperature=0.8,
	top_p=0.95,
	pad_token_id=tokenizer.pad_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Limitations

- The model is tiny and heavily capacity-limited.
- It is trained only on synthetic TinyStories text.
- It is not instruction tuned.
- It is not safety tuned.
- It can repeat, contradict itself, or produce malformed story fragments.
- It should be used for experimentation and testing, not production.

## Data and License Notes

The training dataset card lists TinyStories under `cdla-sharing-1.0`. This model was trained from scratch; it does not contain Gemma weights from Google or weights from TinyLLama-v0.

Weights are released under the license declared in the metadata above. Users are responsible for checking whether their intended use is compatible with the dataset license and applicable law.

## Inspiration

This project was inspired by `Maykeye/TinyLLama-v0`, but uses a Gemma 4 text configuration instead of Llama.