tntholley/zophiae-strawweight

ZophiaE Strawweight

A 25-million-parameter logical-verdict engine that knows when it doesn't know. Byte-level (256-symbol vocabulary, no tokenizer), trained on a single consumer GPU in under two hours total, runs on CPU in pure Rust. Strawweight is the smallest class in the ZophiaE line — named the way boxing names its divisions, and it punches accordingly.

What this is, and why it's built this way

Most language models spend enormous parameter budgets learning what words are — a tokenizer vocabulary, embeddings for tens of thousands of subword fragments. This line takes the opposite bet: words are encoded before training, as concept bytes — a dictionary-defined encoding where one byte is (roughly) one word-meaning and grammar is explicit feature bytes. The model's 25M parameters are spent on reasoning over meanings, not on reconstructing what "bread" is from fragments. Three consequences, all deliberate:

• It's small and it's local. No tokenizer, no cloud, no Python required — the whole thing runs on a CPU, in a folder, offline. This is a bet that useful reasoning doesn't need a datacenter.
• It's inspectable. Every byte in the model's context is a labeled concept, so attention weights are readable relationships, not opaque token indices. The Talkit runtime shows you, per generated byte, what the model weighed and what it attended to.
• It's a specialist on purpose. The training corpus is synthetic, oracle-checked logic, balanced so that refusing to answer is a first-class move (25% of training answers are "we can not say" — and they're correct refusals, not hedges). The result is the rarest property in small models: calibration — it refuses with 98% precision instead of confabulating.

The custom architecture and custom encoding are why this isn't a transformers-loadable model — and also why the whole stack (translator, codec, dictionary, runtime, training script) ships in this repo. Nothing here depends on anything you can't read.

Measured on 2,024 fresh questions proven by hash never to have appeared in any training document:

four-way verdict accuracy (YES / NO / UNKNOWN / OPEN)	89.1% (floor for any constant answerer: 25%)
refusal precision — when it says "we can not say," it's right	98.1%
answers that close their own frame (END discipline)	~99%
the same architecture at half the data ration	35.9%

It computes — real division, real counting; change a number in your question and the verdict follows the arithmetic, not the template. It refuses when premises genuinely underdetermine the answer (denying the antecedent is a refusal here, not a yes). And it shows its work in every answer.

Q: john has 56 breads. the group uses 4 breads each month. the bread
   must last 8 months. if the bread does not last, then the sun sets.
   does the sun set?

A: 56 / 4 = 14. 14 months against 8, so the bread lasts. If the bread
   does not last, then the sun sets. The bread lasts. The rule speaks
   only about what happens when the bread does not last. We can not
   say. The words can all be true while the sun sets, and also while
   the sun does not set.

That is the correct answer — and the branch most models fail.

How to run it

This is not a transformers / GGUF model — the architecture and the concept-byte codec are custom. So this repo ships the entire stack: weights, dictionary, the English↔bytes translator, the reference PyTorch implementation, and the signed Windows appliance. Three ways in, all self-contained:

1. Windows, zero install. Download the repo, run SETUP_TALKIT.bat once (it arranges the folders), then double-click talkit.exe. Chat window, reasoning pane with per-byte attention, a Verify tab that reproduces the numbers above on your machine, and a model picker. (Also on GitHub.)

2. Pure Python (pip install torch safetensors — nothing else):

python run_strawweight.py "a beetle is an insect. all insects are
    animals. so the beetle is an animal. if the beetle is an
    animal, then the horse sleeps. does the horse sleep?"

Runs translator → codec → model → decode → render and prints the annotated stream plus the English answer. train_25m.py here is the actual, unabridged training script — the model definition and the full recipe in one file. zofiae.py is the codec, simplify.py the translator, data/table/ the dictionary.

3. As a local service:

talkit serve --port 8484
POST /ask {"question":"..."}  ->  {"answer","kind","confidence_mean",...}

Questions.md is the playbook: every trained question family with paste-ready examples.

It's a pretrain — continue training on your data

Strawweight is a base model. The intended use is to take it, continue training on your data, and see whether it works for your problem. Everything needed is in this repo, and the loop has been run end to end exactly as written below:

1. Make a corpus from your text (no Python): drop files or paste text into Talkit's Build Corpus tab, or headless —

talkit corpus C:\path\to\your_text_folder     -> your_text_folder\corpus.bin

Each file or paste becomes one concept-byte record through the strict translator (a sentence either survives round-trip-perfect or is dropped with its reason). Keep records under 2,560 bytes — a paragraph, or one question-and-answer — the trainer refuses longer documents. Q&A-framed records (question, then the answer) come from the chat→corpus button; plain records train as ordinary text.

2. Continue training from the published weights (pip install torch safetensors):

set ZOFIAE_MODEL=25m
set ZOFIAE_CORPUS=C:\path\to\your_text_folder
set ZOFIAE_RUNS=C:\path\to\runs
python train_25m.py --resume model.safetensors --corpus-glob "corpus.bin" --epochs 3 --tag mine

train_25m.py resumes directly from this repo's model.safetensors and writes runs\run_NNN_mine\best.pt. With a small corpus the held- out split will be empty (eval_seen 0 docs) — that is expected; judge the result with your own questions or the Verify tab, not that line.

3. Put the result back into Talkit

set TALKIT_ASSETS_DIR=C:\path\to\Talkit\assets
set TALKIT_NAME=my fine-tune
python export_talkit_assets.py C:\path\to\runs\run_001_mine\best.pt my-model

It lands in assets\models\my-model\, the picker shows it on next launch, and TALKIT_MODEL=my-model talkit serve serves it. Keep the original folder beside it and use A/B to compare before and after.

License note: a model continued from these weights is a modified version of the Materials, not an Output — it carries the same license (free noncommercial, commercial by written license capped at 6%). See LICENSE.md §4.

Watching it think, reading the weights

Interpretability isn't a research rig bolted onto this model — it's a property of the encoding. Every position in the context is a labeled dictionary concept, so attention weights are readable relationships and the learned space can be inspected by name. Four instruments, all included:

1. The Talkit reasoning pane (no code): click any answer bubble — per generated byte, the candidate distribution the model weighed and the attention it paid, per layer and head. talkit trace "question" dumps the same evidence headless.

2. watch_it_think.py — the Python version, labeled end to end:

   step   5  -> 4    | weighed: 4 100%  7 0%  5 0%   | looked at: / 0.23, last 0.06, 56 0.04
   step   9  -> 1    | weighed: 1 100%  2 0%  9 0%   | looked at: = 0.13, / 0.11, 56 0.08
   

   That's the model doing the division — while it writes "14", its attention sits on /, 56, 4, and =. Attention is recomputed explicitly (fused kernels don't return weights), mirroring the training forward line for line.

3. read_weights.py — composes every dictionary word through the network once and lets you cosine-explore the learned space by name: sun sits with snow, hot, dark; rise with appear and climb. No probing classifiers — the labels were built into the vocabulary before training.

4. train_25m.py's own instrumentation — the training log reports a frequency-to-attention correlation each epoch: if attention merely mirrored token frequency, the model would be counting, not reasoning. (v3's stayed near zero: r ≈ +0.08.)

A worked example of what this catches: asked about a sun that "sets" — a verb never trained in that slot — the model restated the rule as "the sun falls," the semantically correct pick out of seven trained sky verbs, rejecting the frequency-favorite "rises." watch_it_think.py shows the decision being made in context; read_weights.py shows the labeled space it was made in.

The box it lives in (honest scope)

Strawweight is a specialist: eleven trained families of logic — propositional rules, syllogism chains, temporal/interval reasoning, quantities and proportions, inclusion–exclusion counting, evidence vetting, defaults and exceptions, board tactics, invariants and reachability, proof patterns, and two-family composition chains of the above. English in, through a controlled-language translator that drops what it can't carry, with a stated reason, rather than guessing. Constraint-satisfaction puzzles are its known weak family (~50%). It is not a chatbot; vocabulary is bounded by its dictionary. Inside the box: calibrated. Outside the box: the verdict chip and confidence numbers are the warning instruments.

Architecture

parameters	25,377,152
vocabulary	256 raw bytes — no tokenizer, no BPE
context	2,560 bytes
dims	d_model 384 · 16 layers · 6 heads over 2 KV (GQA 3:1) · FF 1024 (SwiGLU)
position	RoPE base 10000, interleaved pairs
norms	RMSNorm (eps 1e-6) + per-head RMS QK-norm before RoPE
head	untied
precision	fp32 safetensors (~101 MB); runs on CPU

The input is not text bytes: words are concept bytes — a dictionary-defined encoding where one byte is (roughly) one lemma and grammatical features are explicit feature bytes. The dictionary ships with the runtime as four readable TSVs.

Training recipe (the part that mattered)

Two phases, each sized to exactly 10 encoded bytes per parameter ("neochinchilla"), one epoch each, on one RTX 4070 Ti SUPER:

1. Logic school — 253.8 MB of synthetic documents from oracle-checked generators, balanced to 25% each of YES / NO / UNKNOWN / OPEN per pattern, so no constant responder can beat chance; half single-family documents, half two-family composition chains. Every document seen exactly once.
2. Association — 90% curated prose + everyday sentences, with 10% freshly generated logic interspersed throughout (disjointness from phase 1 proven by hash, not hoped from seeds). Held-out logic loss improved through this phase: 0.1769 → 0.1712.

The measured headline: the same architecture trained on half the ration with a two-way corpus scored 35.9% on the same probe. Data ration and corpus design were the levers; parameters were not. Scale is for fluency.

Files

• model.safetensors — fp32 weights, flat names (emb.weight, blocks.{i}.*, nf.g, head.weight), row-major [out, in]
• config.json — dims + provenance; the runtimes read this
• talkit.exe + SETUP_TALKIT.bat — the signed Windows appliance (Authenticode: TNT Holley, Inc.) and its one-click layout script
• run_strawweight.py — pure-Python inference, end to end
• watch_it_think.py — per-byte candidates + attention, labeled
• read_weights.py — explore the learned concept space by name
• train_25m.py — the unabridged training script = the model definition (set ZOFIAE_MODEL=25m, the default); --resume takes this repo's model.safetensors directly — continue training here
• export_talkit_assets.py — turn a training checkpoint into a Talkit model folder (TALKIT_ASSETS_DIR, see the pretrain section)
• zofiae.py / simplify.py / universal.py — codec, translator, shared math block
• data/table/ — the dictionary (readable TSVs)
• Questions.md — the question-format playbook
• LICENSE.md — Holley Community License 1.0 (worldwide text)

License

Free for noncommercial use — research, learning, personal projects, evaluation, charities, schools, public institutions. Commercial use requires a license (travis@tntholley.com), with a published permanent cap: never more than 6% of gross revenue attributable to the licensed use. Using the materials constitutes acceptance of the terms — see LICENSE.md.

Citation

@software{holley2026strawweight,
  author  = {Holley, Travis E.},
  title   = {ZophiaE Strawweight: a 25M-parameter calibrated
             logical-verdict engine on concept bytes},
  year    = {2026},
  url     = {https://github.com/Laninthalesdran/Concept-as-Byte},
  note    = {Weights: huggingface.co/tntholley. Patents pending.}
}

TNT Holley, Inc. · Fort Mill, South Carolina · Patent pending.