🏯 Edo-Cutoff LLM
A 1B-parameter language model that has never read a single word written after 1868.
江戸カットオフLLM — 知識カットオフ 1868年
1. Introduction
Modern LLMs know everything humanity has ever written. Edo-Cutoff LLM inverts that premise.
This model is pretrained exclusively on public-domain texts from the Edo period and earlier — Japanese woodblock-print books and the Chinese classics an educated person of that era would actually have read. It has never seen a modern webpage, a modern textbook, or a modern sentence. It does not know that the Earth orbits the Sun.
The interesting engineering problem is not building such a model — it is preventing contamination. Any modern text, any modern paraphrase, any modern tokenizer leaks 21st-century priors into the weights. This project therefore uses:
- No modern text. Only pre-1868 public-domain sources.
- No modern-model synthetic data. No rephrasing, no distillation.
- A tokenizer trained from scratch on the period corpus (a modern tokenizer's vocabulary is itself a modern prior — and 2× less efficient here).
これは何か: 明治より前の世界しか知らないAI。現代テキストを一切混ぜず、江戸期以前のパブリックドメイン文献のみで事前学習しています。企画の核心は「現代知識の汚染をいかに防ぐか」です。
2. Model Summary
| Architecture | Llama 3.2 1B (decoder-only, GQA, RoPE, SwiGLU, RMSNorm) |
| Parameters | 1.07 B (1,071.5 M) · embeddings 98.3 M (tied) |
| Layers / Hidden / FFN | 16 / 2048 / 8192 |
| Attention heads | 32 query · 8 key-value (GQA 4×, head dim 64) |
| Positional encoding | RoPE (θ = 500,000) |
| Context length | 2,048 |
| Vocabulary | 48,000 (custom byte-level BPE) |
| Precision | bfloat16 |
| Pretraining tokens | 2.29 B corpus · 262 M seen (step 500 / 4,370) |
| Language | Classical Japanese (文語・候文), Literary Chinese (漢文) |
3. Training Data
Everything is public domain by age, and re-distributable. No modern text is included.
| Source | Scale | Content | License |
|---|---|---|---|
| NDL Next Digital Library — 古典籍 OCR | 64,769 books · 2.49 B chars | Japanese woodblock books: literature, medicine, agriculture, travel, Buddhism, Confucian studies | Free reuse (copyright expired) |
| Kanseki Repository (KR1–5 經史子集叢) | 3,943 texts · 0.78 B chars | Chinese classics read in Edo Japan: Four Books, Records of the Grand Historian, philosophers, poetry | CC BY-SA 4.0 |
| Total | 3.27 B chars ≈ 2.29 B tokens | CC BY-SA 4.0 |
Notes on source selection. ctext.org was rejected despite covering the same public-domain works: its data is CC BY-NC-SA and bulk access is gated. The Kanseki Repository provides the same corpus under BY-SA with unrestricted bulk access. The Buddhist canon (KR6, 4,849 texts) was excluded because it is CBETA-derived and NC-encumbered — a real loss, since Edo scholars read it widely.
4. Tokenizer
A byte-level BPE trained from scratch on the period corpus. This is the single highest-leverage component: for a small model with a large vocabulary and long context, tokenizer efficiency directly buys training throughput and effective context.
| Tokenizer | Vocab | Japanese ↓ | Literary Chinese ↓ |
|---|---|---|---|
| Edo-mixed BPE (this work) | 48,000 | 0.611 | 0.836 |
| Edo JP-only BPE | 32,000 | 0.673 | 0.935 |
GPT-4o (o200k_base) |
~200,000 | 1.040 | 1.178 |
GPT-4 (cl100k_base) |
~100,000 | 1.474 | — |
Fertility = tokens per character; lower is better. Measured on held-out Edo-period text.
~1.7× more efficient than GPT-4o and 2.4× more than GPT-4 on this domain. Byte-level construction means zero UNK by design — variant kana (変体仮名), rare glyphs, and kanbun are all representable. Normalization is NFC only; NFKC was rejected because it collapses the orthographic variants that define this corpus.
5. Training
| Hardware | 1 × NVIDIA H100 80GB (RunPod, secure cloud) |
| Stack | PyTorch 2.8 · CUDA 12.8 · bf16 · Liger Kernel (fused cross-entropy) |
| Optimizer | AdamW (β = 0.9/0.95, wd 0.1, fused; decay on 2-D params only) |
| Schedule | cosine 3e-4 → 3e-5, warmup, grad-clip 1.0 |
| Batch | 524,288 tokens/step (256 × 2048), gradient checkpointing |
| Throughput | ~46–55 K tokens/s |
| Progress | step 500 / 4,370 (≈ 11 % of the first epoch) |
Fused cross-entropy is what makes this tractable: with a 48 K vocabulary at 2048 context, the logit tensor dominates activation memory and caps the micro-batch. Liger removes it, roughly doubling throughput over the naive configuration.
Checkpoints are pushed to the Hub every 100 steps, making the run resumable across ephemeral GPU rentals — training proceeds in whatever increments the budget allows.
6. Samples
Generated at step 500 (temperature 0.85, top-k 40). The model is deeply undertrained; these are stylistic fragments, not coherent prose.
| Prompt | Continuation |
|---|---|
天保十五年 |
天保十五年九月上直澄良直広直盛直業等公重隆重実永茂信安従五位下 従四位下 正五位下 従四位上…從二位忠久男直兼直輔母式部右衛門通方… |
此薬 |
此薬種数ヲ変ス是等ノ理ヲ得ルハ是等ノ法ヲ以テ理トセンコトハナリ是其ノ大品ニ理ハ別ニ定シ難キ者ニシテ是ノ如ク論スルノミナリ |
子曰 |
子曰然 |
The model has already internalized period-specific registers: court-rank genealogies (武鑑・系図体) from the first prompt, kanbun-kundoku expository prose (漢文訓読体) from the second, and Analects phrasing from the third. Learning the form of Edo writing precedes learning its content.
7. Quick Start
Colab (free GPU, public link)
Open edo_colab_demo.ipynb → set runtime to T4 GPU → Run all. A Gradio app with a public *.gradio.live URL launches automatically.
Python
import torch
from huggingface_hub import hf_hub_download
from transformers import LlamaConfig, LlamaForCausalLM
from tokenizers import Tokenizer
REPO = "zary0/edo-cutoff-1b"
ck = torch.load(hf_hub_download(REPO, "edo_1b_partial_bf16.pt"), map_location="cpu")
tok = Tokenizer.from_file(hf_hub_download(REPO, "edo_bpe_48000.json"))
cfg = LlamaConfig(**{k: v for k, v in ck["config"].items() if not k.startswith("_")})
model = LlamaForCausalLM(cfg)
model.load_state_dict(ck["model"], strict=False)
model = model.to("cuda").to(torch.bfloat16).eval()
ids = [cfg.bos_token_id] + tok.encode("天保十五年").ids
x = torch.tensor([ids], device="cuda")
for _ in range(80):
logits = model(input_ids=x).logits[:, -1, :].float() / 0.85
v, _ = torch.topk(logits, 40)
logits[logits < v[:, [-1]]] = -float("inf")
x = torch.cat([x, torch.multinomial(torch.softmax(logits, -1), 1)], dim=1)
print(tok.decode(x[0].tolist()))
Resume pretraining
hf download zary0/edo-cutoff-1b ckpt_last.pt --local-dir ckpt # full state (12.9 GB, incl. optimizer)
python scripts/train.py --config configs/edo_1b.json --liger --grad-ckpt \
--device cuda --dtype bf16 --resume ckpt/ckpt_last.pt --hf-repo zary0/edo-cutoff-1b
Files
| File | Size | Purpose |
|---|---|---|
edo_1b_partial_bf16.pt |
2.3 GB | Inference weights (bf16) |
ckpt_last.pt |
12.9 GB | Full training state (weights + optimizer), for resuming |
edo_bpe_48000.json |
4 MB | Tokenizer |
edo_colab_demo.ipynb |
— | One-click Colab demo |
8. Limitations
- Severely undertrained. 262 M of a planned 2.29 B tokens (11 % of one epoch). It cannot yet hold a coherent sentence. Training halted purely by compute budget, not by design.
- OCR noise is learned.
□marks illegible characters in the source scans; the model reproduces them as a frequent token. - Not instruction-tuned. It is a raw base model — a continuation engine, not an assistant. It will not answer questions.
- Not purely Japanese. ~24 % literary Chinese, deliberately: that reflects an Edo reader's actual diet.
- The cutoff is a data property, not a guarantee. Contamination was controlled at the source and tokenizer level, but has not been adversarially audited.
9. License
CC BY-SA 4.0, inherited from the Kanseki Repository portion of the training data. Share-alike applies to derivatives.
10. Citation
@misc{edo-cutoff-llm-2026,
title = {Edo-Cutoff LLM: A Language Model with an 1868 Knowledge Cutoff},
author = {zary0},
year = {2026},
url = {https://huggingface.co/zary0/edo-cutoff-1b}
}
Acknowledgements
Built on public infrastructure that made a project like this possible for one person:
- National Diet Library (NDL Lab) — Next Digital Library, and NDL古典籍OCR for kuzushiji recognition (CC BY 4.0)
- Kanseki Repository 漢籍リポジトリ — Christian Wittern and contributors
- みんなで翻刻 — crowd-sourced transcription (CC BY-SA 4.0)