README.md

---
license: apache-2.0
language:
- en
library_name: transformers
tags:
- text-generation
- llama
- pretraining
- from-scratch
pipeline_tag: text-generation
---

# Dot-125M

Dot-125M is a 125M-parameter (133.7M actual), Llama-3-style decoder-only transformer,
pretrained **entirely from scratch** — no fine-tuning or continued pretraining from an
existing checkpoint — by **[Perletter](https://perletter.com)**, part of Chirping Waves
Limited (Ireland).

It was trained on 2.0B tokens (4 epochs over a 500M-token filtered, deduplicated sample
of [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)) on a single
consumer laptop GPU.

This is a **base (pretrained) language model** — it has not been instruction-tuned,
RLHF'd, or chat-templated. It completes text; it does not reliably follow instructions
or hold a conversation.

## Model details

| | |
|---|---|
| Parameters | 133.7M |
| Architecture | Llama-3-style decoder-only, GQA, RoPE, RMSNorm, SwiGLU, tied embeddings |
| Layers / heads / KV heads | 12 / 12 / 4 |
| Hidden size | 960 |
| Context length | 512 |
| Vocab size | 16,384 (byte-level BPE, trained from scratch on the training split) |
| Training tokens | 2.0B (4 epochs × 500M-token corpus) |
| Training data | [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (`sample-10BT`), quality-filtered + exact/near-deduplicated, English only |
| License | Apache 2.0 |

## Benchmarks

Compared against GPT-2-small (124M, ~10B training tokens) — see full write-up for
methodology.

**Bits-per-byte (primary metric, tokenizer-fair comparison), on a held-out test split:**

| | bpb | ppl |
|---|---|---|
| **Dot-125M** | **1.0142** | 20.64 |
| GPT-2-small | 1.0281 | 27.19 |

**lm-evaluation-harness:**

| task | Dot-125M | GPT-2-small |
|---|---|---|
| arc_easy (acc) | 48.23% | 43.81% |
| hellaswag (acc_norm) | 30.86% | 31.14% |
| piqa (acc) | 61.43% | 62.89% |
| winogrande (acc) | 49.57% | 51.62% |
| lambada_openai (acc) | 23.02% | 32.56% |

Mixed on the individual benchmark tasks (stronger on arc_easy, weaker on
lambada_openai's long-range prediction — expected given the token/context budget:
Dot-125M saw 500M unique tokens across 4 epochs at 512 context vs. GPT-2's ~10B tokens
single-pass at 1024 context), but wins on the primary bits-per-byte metric.

**Quantization** (GGUF, via llama.cpp): Q8_0 stays within 0.01% bpb of full-precision
f16. Q4_K_M is available but falls back to a different quant scheme for most tensors
(this model's hidden size isn't a multiple of 256, the k-quant block size) — still only
~0.17% bpb degradation vs. f16, but not "true" Q4_K_M. Q8_0/Q4_0/Q5_0/Q5_1 are the
quant types this model size supports natively.

## Intended use

Research, experimentation, and demonstration of from-scratch small-LM pretraining. Not
instruction-tuned — do not expect chat-assistant behavior out of the box. Not suitable
for production use requiring factual reliability, safety filtering, or instruction
following without further fine-tuning.

## How to use

**Transformers (safetensors):**
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("perletter/Dot-125M")
model = AutoModelForCausalLM.from_pretrained("perletter/Dot-125M")

inputs = tok("The history of the internet", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50)
print(tok.decode(out[0], skip_special_tokens=True))
```

**llama.cpp (GGUF):**
```bash
llama-cli -m model-Q8_0.gguf -p "The history of the internet" -n 50
```

## Limitations

- Small model, small training budget — general knowledge and reasoning are limited
  compared to larger contemporary models.
- English only.
- Base model only — no safety fine-tuning, no RLHF, no instruction-tuning. It will
  complete harmful, biased, or false text if prompted toward it, the same as any
  unaligned base LM.
- 512-token context window.

## License

Apache 2.0 — the model weights are freely available for any use, including commercial,
with no attribution requirement beyond the license notice. See `LICENSE`.

The training code/pipeline used to produce this model is **not** included in this
release.

## Citation

```
@misc{dot125m2026,
  title  = {Dot-125M},
  author = {Perletter, part of Chirping Waves Limited},
  year   = {2026},
  url    = {https://perletter.com}
}
```