Tiamat-base

Tiamat-base is a small byte-level Baby Dragon Hatchling-style language model proof of concept.

This is a custom PyTorch checkpoint, not a Hugging Face Transformers checkpoint. The repository includes the model implementation and inference script used to load it.

Checkpoint

• Recommended file: tiamat-base-step240000-inference.pt
• Training step: 240000
• Parameters: 50,462,720
• Vocabulary: raw UTF-8 bytes, 0-255
• EOS token: not used in this checkpoint
• Online plasticity: enabled through the model's carried Hebbian state during sequence processing
• Note: the published checkpoint is inference-only and does not include optimizer, scheduler, or dataloader state.

Architecture

n_layer: 6
n_embd: 256
n_head: 4
n_neuron: 65536
vocab_size: 256
rope_theta: 65536
state_decay: 0.995
state_clip: 2.0
attention_chunk_size: 64

Usage

Clone or download this repository, install requirements, then run:

python infer.py \
  --checkpoint tiamat-base-step240000-inference.pt \
  --prompt "In this chapter, we describe " \
  --max-new-tokens 300 \
  --temperature 0.8 \
  --top-k 40

Greedy decoding is supported with:

python infer.py \
  --checkpoint tiamat-base-step240000-inference.pt \
  --prompt "In this chapter, we describe " \
  --max-new-tokens 200 \
  --temperature 0.0

Status

This is an early base checkpoint intended for experimentation. It can produce mostly valid byte-level English-like continuations, but it is not instruction-tuned and should not be treated as reliable.

Later Tiamat fine-tuned checkpoints may use richer datasets and may add an explicit EOS token, which would change the vocabulary size and checkpoint compatibility.