| --- |
| license: cc-by-sa-4.0 |
| language: |
| - la |
| library_name: onnx |
| pipeline_tag: text-generation |
| tags: |
| - latin |
| - gpt |
| - from-scratch |
| - onnx |
| - classical-latin |
| --- |
| |
| # Cicero LLM |
|
|
| A 100M-parameter Latin language model, **trained from scratch** — no pretrained |
| backbone, no English/Greek base. It generates Classical Latin in the browser |
| or anywhere ONNX runs. |
|
|
| Live demo (browser inference): https://cicerollm.com |
|
|
| ## Model |
|
|
| - Decoder-only transformer, ~111M params (12 layers × 12 heads × 768 dim, |
| 2048 block size, learned absolute positions, tied embeddings) |
| - 32K SentencePiece-BPE tokenizer trained on the same Latin corpus |
| - Trained from random init on a ~466M-token Latin corpus (30,000 steps, |
| dropout 0.15), then **continued-pretrained on a targeted classical-grammar |
| curriculum** (synthetic Cicero-register prose, generated and quality-filtered |
| by a stronger model) mixed 30/70 with clean classical replay for 3,000 steps. |
| The curriculum step pushes generation toward classical register and cuts the |
| medieval/neo-Latin contamination and repetition of the base model. |
|
|
| ## Evaluation |
|
|
| Cloze accuracy (4-option multiple choice; held-out "blind" pack is the honest |
| cross-model number): |
|
|
| | pack | accuracy | |
| |---|---| |
| | held-out blind (144 items) | 0.72 | |
| | literary diagnostic | 0.82 | |
| | grammar-probe / weakness (60 items) | 0.82 | |
| | in-distribution textbook | 0.77 | |
| | bits-per-char (held-out) | 1.56 | |
|
|
| ## Files |
|
|
| - `model.int8.onnx` — int8-quantized ONNX (~136 MB; used by the browser demo) |
| - `model.onnx` — fp32 ONNX (~543 MB) |
| - `checkpoint_step_033000.pt` — raw PyTorch weights + optimizer state (~1.3 GB) |
| - `tokenizer.json`, `tokenizer.model`, `tokenizer_config.json` — SentencePiece 32K |
| - `config.json` — architecture metadata |
|
|
| ## Usage (ONNX Runtime) |
|
|
| ```python |
| import onnxruntime as ort, numpy as np, sentencepiece as smp |
| sp = smp.SentencePieceProcessor(model_file="tokenizer.model") |
| sess = ort.InferenceSession("model.int8.onnx") |
| ids = sp.encode("Gallia est omnis divisa", out_type=int) |
| # forward returns next-token logits at the last position; sample autoregressively |
| logits = sess.run(None, {"input_ids": np.array([ids], dtype=np.int64)})[0] |
| ``` |
|
|
| ## Limitations |
|
|
| Research artifact. Autoregressive completion with temperature + top-k sampling; |
| no instruction tuning, no chat behavior. Give it Latin and it continues in |
| Latin. Best results in classical (Caesarian / Ciceronian) register. |
|
|
| ## License |
|
|
| CC-BY-SA-4.0. The underlying ancient texts are public domain by age; the |
| share-alike condition derives from corpus components (e.g. Perseus digital |
| editions). Attribution + share-alike apply to redistribution. |
|
|
