README.md

metadata

license: cc-by-sa-4.0
language:
  - la
library_name: onnx
pipeline_tag: text-generation
tags:
  - latin
  - gpt
  - from-scratch
  - onnx
  - classical-latin

Cicero LLM

A 100M-parameter Latin language model, trained from scratch — no pretrained backbone, no English/Greek base. It generates Classical Latin in the browser or anywhere ONNX runs.

Live demo (browser inference): https://cicerollm.com

Model

• Decoder-only transformer, ~111M params (12 layers × 12 heads × 768 dim, 2048 block size, learned absolute positions, tied embeddings)
• 32K SentencePiece-BPE tokenizer trained on the same Latin corpus
• Trained from random init on a ~466M-token Latin corpus (30,000 steps, dropout 0.15), then continued-pretrained on a targeted classical-grammar curriculum (synthetic Cicero-register prose, generated and quality-filtered by a stronger model) mixed 30/70 with clean classical replay for 3,000 steps. The curriculum step pushes generation toward classical register and cuts the medieval/neo-Latin contamination and repetition of the base model.

Evaluation

Cloze accuracy (4-option multiple choice; held-out "blind" pack is the honest cross-model number):

pack	accuracy
held-out blind (144 items)	0.72
literary diagnostic	0.82
grammar-probe / weakness (60 items)	0.82
in-distribution textbook	0.77
bits-per-char (held-out)	1.56

Files

• model.int8.onnx — int8-quantized ONNX (~136 MB; used by the browser demo)
• model.onnx — fp32 ONNX (~543 MB)
• checkpoint_step_033000.pt — raw PyTorch weights + optimizer state (~1.3 GB)
• tokenizer.json, tokenizer.model, tokenizer_config.json — SentencePiece 32K
• config.json — architecture metadata

Usage (ONNX Runtime)

import onnxruntime as ort, numpy as np, sentencepiece as smp
sp = smp.SentencePieceProcessor(model_file="tokenizer.model")
sess = ort.InferenceSession("model.int8.onnx")
ids = sp.encode("Gallia est omnis divisa", out_type=int)
# forward returns next-token logits at the last position; sample autoregressively
logits = sess.run(None, {"input_ids": np.array([ids], dtype=np.int64)})[0]

Limitations

Research artifact. Autoregressive completion with temperature + top-k sampling; no instruction tuning, no chat behavior. Give it Latin and it continues in Latin. Best results in classical (Caesarian / Ciceronian) register.

License

CC-BY-SA-4.0. The underlying ancient texts are public domain by age; the share-alike condition derives from corpus components (e.g. Perseus digital editions). Attribution + share-alike apply to redistribution.