TevunahAi/Chochmah1B

Chochmah-1B - TevunahAi From-Scratch Pretrained Base Model

This is our second from-scratch model and the 1B promised in the Chochmah-350M card. Same formula, more of it: the same hand-built case-law, philosophy, and textbook corpora, a new hand-built narrative corpus, 45 billion tokens instead of 30, and a 1.18B-parameter model with grouped-query attention and QK-norm in place of the plain Llama block. Still one workstation GPU. The two data defects we documented on the 350M — hard line wraps in the Gutenberg text and fiction leaking into the philosophy corpus — were fixed before this run started.

Chochmah (חָכְמָה, wisdom) is a 1.18B-parameter decoder-only language model pretrained from scratch by TevunahAi on 45 billion tokens, on a single NVIDIA RTX 5000 Ada (32 GB), in 44 days of wall-clock time. It is the companion term to tevunah (understanding, discernment), from which TevunahAi takes its name; the two appear side by side in Proverbs 3:19.

It is a base model: no instruction tuning, no chat template, no RLHF. The pretraining mix was deliberately weighted toward case law, philosophy and political philosophy, mathematics, physics, and code — using four hand-built corpora (2.6 million US judicial opinions, a curated public-domain philosophy canon, 24,175 Gutenberg works of fiction, OpenStax textbooks) on top of a broad web / code / encyclopedia foundation, trained through a three-phase curriculum aligned to a warmup–stable–decay learning-rate schedule.

It is released as (a) a clean, fully documented ~1B base model for fine-tuning and quantization research, and (b) a complete worked example of a from-scratch pretraining pipeline — data curation, tokenization, curriculum, schedule, crash-safe training — that fits on one workstation GPU.

On the architecture name: this model loads as Qwen3ForCausalLM because its block (GQA, per-head QK-norm before RoPE, SwiGLU, RMSNorm, tied embeddings) is exactly the Qwen3 block, which means every tool that supports Qwen3 supports this model with no custom code. It is the architecture only. No Qwen weights, data, or tokenizer were used; every parameter was trained from random initialization on the data described below.

What Changed From Chochmah-350M

	Chochmah-350M	Chochmah-1B
Parameters	358.7M	1,182.9M
Training tokens	30B	45B
Context length	2,048	4,096
Attention	MHA, 16 heads, head dim 64	GQA 16 Q / 4 KV heads, head dim 128, QK-norm
Hidden / intermediate	1,024 / 2,816	2,048 / 5,632
RoPE theta	10,000	50,000
Global batch	524,288 tokens	1,048,576 tokens
Foundation books	raw Gutenberg, hard-wrapped	hand-built narrative corpus, unwrapped
Philosophy corpus	3,072 works, fiction leakage	2,693 works, de-duplicated against fiction, unwrapped
Law share in phase 2	18.8%	27.8%
Architecture class	LlamaForCausalLM	Qwen3ForCausalLM
Wall-clock	10.5 days	44 days

Model Details

Property	Value
Developer	TevunahAi
Model Type	Decoder-only transformer, Qwen3-architecture block, pretrained from scratch
Parameters	1,182.9M total (1,082.2M non-embedding)
Training Tokens	45.00B (42,915 steps x 1,048,576 tokens)
Context Length	4,096 tokens
Tokenizer	SmolLM2 (49,152 BPE)
Training Precision	bf16 autocast, fp32 master weights
Release Precision	fp32 safetensors (4.73 GB)
Training Hardware	1x NVIDIA RTX 5000 Ada 32 GB
Training Time	44 days wall-clock (August 26 – October 9, 2026)
Training Compute	~3.2 x 10^20 FLOPs
Stage	Base — pretrained only
Loads With	AutoModelForCausalLM / Qwen3ForCausalLM, transformers >= 4.51, no custom code
License	Apache-2.0

Architecture Breakdown

A modern decoder with two deliberate upgrades over the 350M: grouped-query attention, so the 4,096-token context is cheap to serve, and QK-norm, so attention stays stable through the whole decay. Everything else is the current default so the model loads everywhere Qwen3 loads.

Layer Composition (24 layers)

• 24 Transformer Blocks, pre-norm residual
  • Grouped-Query Attention: 16 query heads, 4 key/value heads, head dimension 128, no biases
  • QK-norm: RMSNorm over the head dimension applied to Q and K before RoPE
  • SwiGLU MLP: intermediate size 5,632, down(silu(gate(x)) * up(x)), no biases
  • RMSNorm before attention and before the MLP, eps 1e-5
• Rotary position embeddings (RoPE), theta 50,000, applied to Q and K
• Tied input/output embeddings: the 49,152 x 2,048 embedding matrix is also the LM head
• No dropout, no bias terms anywhere
• z-loss (1e-4) on the logits during training only; it leaves no trace in the weights

Parameter Accounting

Component	Parameters
Attention (24 x (2 x 2048^2 + 2 x 512 x 2048))	251.7M
SwiGLU MLP (24 x 3 x 2048 x 5632)	830.5M
Norms, including QK-norm	~0.1M
Non-embedding total	1,082.2M
Tied embedding (49,152 x 2048)	100.7M
Total	1,182.9M

Why This Matters

• Non-embedding count drives compute. 1.08B non-embedding parameters at 45B tokens is why this run took 44 days on one GPU.
• 70% of the parameters are in the MLPs. That is where the model's factual and stylistic knowledge lives, which matters for quantization planning.
• GQA 4:1. The KV cache is a quarter the size of an MHA model of the same width — about 0.4 GB in fp32 at full 4,096 context — so long-context inference fits where the 350M's shorter context did.
• QK-norm. Normalizing Q and K per head bounds the attention logits; the gradient norm held at 0.08–0.09 through the entire learning-rate decay with no instability.
• Qwen3-compatible weights. Conversion from the training implementation is a pure key rename (no weight permutation); logit equivalence between the original PyTorch model and Qwen3ForCausalLM was verified to 2.5e-5 max absolute difference in fp32 with 100% next-token agreement. Everything downstream — vLLM, GGUF, GPTQModel — works out of the box.

Training

Optimizer and Schedule

Field	Value
Optimizer	FILL
Weight Decay	FILL
Peak Learning Rate	FILL
Schedule	WSD (warmup–stable–decay): FILL-step linear warmup → constant through phases 1 and 2 → cosine decay to 0 across the final 4,291 steps (the anneal phase, steps 38,624–42,915)
Gradient Clipping	FILL
Auxiliary Loss	z-loss, coefficient 1e-4
Global Batch	1,048,576 tokens (256 sequences x 4,096)
Total Steps	42,915
Precision	bf16 autocast, fp32 master weights and optimizer state
Kernels	torch.compile, SDPA with native GQA, fused AdamW
Throughput	~10,500 tok/s median, peak VRAM 24.2 GB, ~28% MFU
Checkpointing	Atomic writes (temp + rename) every 30 minutes, full optimizer / RNG / dataloader state; resume verified bit-exact

The learning-rate decay is aligned to the data curriculum on purpose: the LR stays flat while data quality ramps through phases 1 and 2, then decays to zero exactly across the anneal phase. That alignment is what lets the highest-quality 10% of the data set the model's final voice instead of being averaged away by later updates.

Loss trajectory. 11.19 at initialization (matching the predicted loss for a uniform 49,152-way prediction) → FILL entering the anneal (step 38,624) → ~1.54 at the end of training, with the per-step value in a 1.48–1.62 band over the final 80 steps (measured on the anneal mix; the phases use different data, so losses are not directly comparable across boundaries). Gradient norm held at 0.08–0.09 through the entire decay. Both phase transitions occurred automatically without a restart; the one restart in the run, to add evaluation instrumentation, resumed from checkpoint.

Three-Phase Curriculum

Phase	Steps	Tokens	Purpose
1 — Foundation	0 – 25,750	27.0B	Broad language competence. Deliberately unclever.
2 — Domain build	25,750 – 38,624	13.5B	The target domains enter at full weight; law at its highest share.
3 — Anneal	38,624 – 42,915	4.5B	Highest-quality material only, LR decaying to zero. Sets the model's voice.

Each phase was pre-mixed into shards at fixed ratios (no runtime sampling), with a consumption manifest tracking per-source byte offsets and head hashes so that the three phases read disjoint ranges of every source. The manifest was started fresh for this run: nothing inherits an offset from the 350M. Phase 1 holds every source under one epoch (0.91). Only the small hand-built corpora repeat in later phases: philosophy 3.0 epochs, textbooks FILL epochs. Separate per-domain validation sets were carved from the unused tail of each source, with the exception of philosophy, whose validation set is in-sample (see the per-domain table).

Phase 1 — Foundation (27.0B tokens) — all six sources at 0.91 epochs

Source	Share
FineWeb-Edu	FILL
StarCoderData	FILL
Wikipedia (en)	FILL
Narrative (hand-built, Gutenberg fiction)	FILL
OpenWebMath	FILL
Cosmopedia v2	FILL

Phase 2 — Domain build (13.5B tokens)

Source	Share
Law (hand-built, broad)	27.8%
FineWeb-Edu	FILL
Math (FineMath-4+, OpenWebMath, AlgebraicStack)	FILL
StarCoderData	FILL
peS2o (science)	FILL
Cosmopedia v2	FILL
Philosophy (hand-built)	3%
Textbooks (OpenStax)	FILL

Phase 3 — Anneal (4.5B tokens, LR → 0)

Source	Share
Cosmopedia v2	FILL
FineMath-4+	FILL
StarCoderData	FILL
Law canon (hand-built, most-cited opinions)	FILL
Philosophy (hand-built)	10%
FineWeb-Edu	FILL
Textbooks (OpenStax)	FILL

Data Sources

Source	Origin	License	Role
FineWeb-Edu	HuggingFaceFW/fineweb-edu (sample-100BT)	ODC-By	Foundation, all phases
StarCoderData	bigcode/starcoderdata	Permissive-licensed code	Code, all phases
Wikipedia	wikimedia/wikipedia (20231101.en)	CC BY-SA 4.0	Phase 1
OpenWebMath	open-web-math/open-web-math	ODC-By	Math, phases 1–2
Cosmopedia v2	HuggingFaceTB/smollm-corpus	Apache-2.0	Synthetic textbook prose, all phases; anneal anchor
FineMath-4+	HuggingFaceTB/finemath	ODC-By	Math, phases 2–3
AlgebraicStack	EleutherAI/proof-pile-2	Mixed permissive	Math/code, phase 2
peS2o	allenai/peS2o	ODC-By	Science (abstracts + introductions), phase 2
Narrative	sedthh/gutenberg_english, filtered to fiction, unwrapped	Public domain	Hand-built; phase 1
Law	CourtListener bulk export, Free Law Project	Public domain	Hand-built; phases 2 (broad) and 3 (canon)
Philosophy	Project Gutenberg, curated, unwrapped	Public domain	Hand-built; phases 2–3
Textbooks	OpenStax CNXML sources (GitHub)	CC BY 4.0	Hand-built; phases 2–3

The Hand-Built Corpora

Narrative — 24,175 works of fiction (~1.95B tokens). New for this model, replacing the raw Gutenberg dump the 350M trained on. Gutenberg's hard line wraps — the source of the most visible formatting artifact in the 350M's output — were removed with a line-unwrapping pass that leaves paragraph structure intact; after repair, about 1.4% of works still carry line breaks, essentially all of them verse and drama where the breaks are the text. Consumed at 0.91 epochs in phase 1 as foundation material.

Law — 2,556,649 opinions (9.92B tokens) broad, 101,688 opinions (0.80B tokens) canon. Built from the 2026-06-30 CourtListener bulk export. Courts were selected by jurisdiction level, not by name (New York's "Supreme Court" is a trial court): the Supreme Court of the United States, the 13 modern federal circuits, and state courts of last resort; state intermediate appellate courts and the pre-1911 circuit trial courts were excluded. Opinions that were too short (1.6M) or carried West headnote markers were rejected. The canon split — every opinion cited 160 or more times, plus all SCOTUS opinions — is what the anneal phase trained on; it is 13% SCOTUS by token count and 73% from the 1980s–2010s. The broad corpus spans 66 courts with 42% of tokens pre-1980. Law was raised to 27.8% of phase 2 for this model, which still consumed only 0.38 epochs of the broad corpus and FILL epochs of the canon: the law the model saw is a random cross-section, not a skim of the beginning.

Philosophy — 2,693 works (~0.285B tokens), 126 curated authors across seven eras. The 350M's corpus, repaired twice. First, the same line-unwrapping pass as the narrative corpus (80% of lines were hard-wrapped before the fix). Second, an audit against the narrative corpus found 330 works that were fiction admitted through the subject-heading fallback, plus 57 strays (cookbooks, medical texts, translator-attributed Iliads); all 387 were removed. Ancient through early twentieth century, deliberately broadened to political theory and intellectual history, including Eastern texts, Douglass, Du Bois, Spooner, and the public-domain women's canon (Wollstonecraft, Fuller, Stanton, Harriet Taylor Mill, Anna Julia Cooper, Gilman, Goldman, Addams, de Gouges, Martineau). Nietzsche is present in full (22 works, the complete Levy edition), alongside Voltaire (23 works), Marx, Aristotle, Kant, Machiavelli, Confucius; the largest contributors by volume are William James, Hume, Henry George, T. H. Green, and Augustine. Matched by author, then Library of Congress classification, then subject; Gutenberg license boilerplate stripped; duplicate editions collapsed to the longest. Consumed at 3.0 epochs across phases 2 and 3. One data note: the staging pipeline's 8 MB per-document cap dropped a single philosophy document, almost certainly Ingersoll's Complete Works (~2.2M tokens), from the mix.

Textbooks — 19 OpenStax STEM and government titles (~14M tokens). Fetched as CNXML from the openstax/osbooks-* source repositories, stripped of MathML, media, and metadata to clean prose. Openly licensed real textbooks are scarce (~100–300M tokens exist in total), so this corpus supplies authenticity while Cosmopedia supplies textbook-style volume. Consumed at FILL epochs.

Tokenizer

HuggingFaceTB/SmolLM2-360M — 49,152 BPE with digit splitting and code-friendly merges, chosen for a LaTeX- and code-heavy mix. Documents were concatenated with <|endoftext|> as the separator. No BOS token was prepended during training — do not add one at inference.

Performance Benchmarks

Standard Benchmarks

0-shot, lm-evaluation-harness, fp32. All models were evaluated with the identical command on the same machine so the columns are directly comparable. Pythia-1B is the closest compute-matched reference and saw 300B tokens; SmolLM2-1.7B shares this model's tokenizer and saw 11T.

Task	Chochmah-1B (45B tokens)	Chochmah-350M (30B tokens)	Pythia-1B (300B tokens)	SmolLM2-1.7B (11T tokens)
ARC-Easy (acc / acc_norm)	FILL / FILL	59.0 / 54.2	FILL / FILL	FILL / FILL
HellaSwag (acc / acc_norm)	FILL / FILL	34.8 / 41.8	FILL / FILL	FILL / FILL
PIQA (acc / acc_norm)	FILL / FILL	68.6 / 68.3	FILL / FILL	FILL / FILL
LAMBADA-OpenAI (acc / ppl ↓)	FILL / FILL	40.2 / 23.8	FILL / FILL	FILL / FILL

Per-Domain Held-Out Loss

Validation loss (nats/token) on held-out sets carved from the unused tail of each source, measured at step 42,000 of 42,915 — the last scheduled in-run evaluation, 0.96B tokens before the end of the anneal.

Domain	Loss
FineWeb-Edu	2.50
Wikipedia	2.37
Narrative (fiction)	2.70
StarCoderData	0.70
OpenWebMath	1.95
Cosmopedia v2	1.39
FineMath-4+	1.42
AlgebraicStack	0.67
peS2o	2.49
Law (broad)	1.75
Law (canon)	1.72
Philosophy (in-sample)	2.44
Unweighted mean	1.84

The unweighted mean sits above the ~1.54 training loss because the anneal mix leans on the low-entropy sources (code, synthetic textbook prose, case law); weighted by the phase-3 shares, the two agree. Fiction is the highest-loss domain in the mix, which is a property of fiction, not a weakness: it has the least predictable next token of anything here. The philosophy set is in-sample — drawn from documents the model trained on, because the corpus is too small to hold out a tail and was repeated 3.0 epochs — so read it as a memorization check, not a generalization measure. That it sits in the same band as the held-out prose domains rather than well below them says the three epochs did not turn into recitation.

Sample Outputs

Verbatim, unedited, from the first prompts run on the final checkpoint (temperature 0.7, top-k 50, top-p 0.95, 200 new tokens). The first two are shown together deliberately: they are the same model, one prompt apart, and together they are an accurate description of what a 1B base model is.

Prompt: The Supreme Court of the United States has long held that

The Supreme Court of the United States has long held that the Due Process Clause of the Fourteenth Amendment requires the government to provide a fair trial before an impartial jury and not to deny a defendant the right to an impartial jury. See, e.g., Patton v. Yount, 467 U.S. 1025, 1031 , 104 S.Ct. 2885 , 81 L.Ed.2d 847 (1984) (holding that a defendant’s right to be tried by an impartial jury is violated when the jury selection process is tainted by racial bias); United States v. Martinez-Salazar, 528 U.S. 304, 313 , 120 S.Ct. 774 , 145 L.Ed.2d 792 (2000) (“[T]he Sixth Amendment guarantees the defendant the

Both citations are real, correctly reported in all three reporters, and both cases are in fact impartial-jury cases — the citation apparatus of a 1B model trained on 3.75B tokens of case law is in order. The parenthetical for Patton is wrong: that case is about pretrial publicity, not racial bias. The spaces before the commas inside the citations are a CourtListener HTML-to-text artifact the model learned faithfully.

Prompt: In Marbury v. Madison, Chief Justice Marshall reasoned that

In Marbury v. Madison, Chief Justice Marshall reasoned that because the President "is the only person authorized by the constitution to exercise executive power, his acts in that character are as much within the protection of the Constitution, as those of any other individual," 1 Cranch, at 176 . In this case, the President has acted under the authority granted to him by the Constitution. Accordingly, the majority's argument that the President's removal power is "not as broad as that of Congress" is simply incorrect. Ante, at 375.

The majority also argues that if the President has "executive power," then the President's removal power is "not as broad as that of Congress." Ante, at 375. The majority does not explain how this is so. The Court in Marbury v. Madison, in deciding whether the President had "executive power," did not examine whether Congress had "legislative power." Indeed, the Court decided the case on the ground that the President had "

The register is a dissent, cold: "Ante, at 375" is how a dissent cites the majority, and "1 Cranch, at 176" is a real pincite into Marbury. The quotation is invented, and the removal-power argument it is deployed in is not in Marbury at all; it is the shape of a removal-power dissent from a different century. Compared with the 350M, which got the subject of Marbury wrong outright, the 1B gets the subject right and fabricates one level down, inside the quotation marks. That is progress, and it is also more dangerous, because it reads as authoritative. At this scale the model stores forms, not concepts, and the forms are now very good.

More samples — physics, philosophy, and code

Prompt: Newton's second law states that

Newton's second law states that the net force acting on a body is equal to the mass of the body multiplied by its acceleration. Mathematically, this can be expressed as:

F = ma

where F represents the net force, m represents the mass of the body, and a represents its acceleration.

When applied to the context of the opening extract, we see that the force exerted by the machine is being calculated using the formula F = ma, where F is the force, m is the mass, and a is the acceleration. The machine is considered to be in motion, and the acceleration of the machine is being determined by the acceleration of the test mass.

Let's take a closer look at the first part of the equation:

F = ma

This equation states that the net force acting on an object is equal to the product of the mass of the object and its acceleration. In simpler terms, if you have a large mass and a fast acceleration, the force exerted by the

The physics is correct. "The opening extract" is Cosmopedia's fingerprint: its synthetic explainers were generated from a seed snippet and refer back to it, so the model has learned that explainers mention an extract that is not there.

Prompt: Nietzsche's critique of morality begins with the observation that

Nietzsche's critique of morality begins with the observation that morality is a "philosophy of the beautiful." For Nietzsche, this philosophy is a critique of the metaphysical foundations of morality, which he calls the "ethics of the beautiful." According to Nietzsche, the philosophical foundations of morality are built on the idea that morality is a product of reason and that reason alone is capable of creating objective moral values. However, he argues that this idea is false, and that morality is instead a product of emotion and that reason alone is unable to create objective moral values.

Nietzsche's critique of morality is based on his belief that morality is a subjective and arbitrary system of values that is determined by the needs and desires of the individual. He argues that morality is not a natural or universal truth, but rather a product of historical and cultural factors. For example, Nietzsche criticizes the idea of "natural law" and argues that it is not a universal principle that applies to everyone, but rather a political and cultural construct that serves the interests of a particular group of people

"Philosophy of the beautiful" is invented, and the rest is survey-course paraphrase in Cosmopedia's voice rather than anything that sounds like the Genealogy. Nietzsche's ~2M tokens, seen three times, are 0.01% of the run; the model learned the shape of a philosophy explainer far better than the content of the primary texts. Prompting in the register of a primary text rather than a summary about one shifts the output toward the canon's voice.

Prompt: def fibonacci(n): (with docstring)

def fibonacci(n):
    """Return the nth Fibonacci number."""
return fibonacci(n-1) + fibonacci(n-2)
```

### Testing the Function

Now that we have defined our Fibonacci function, let's test it using some sample inputs. We will use the `assert` statement to check that the function raises a `ValueError` if the input is outside the expected range.

```python
def test_fibonacci():
    """Test that fibonacci() correctly returns the nth Fibonacci number."""
    try:
        fibonacci(1)
    except:
        raise ValueError("Input should be in the range 0 <= n <= 20")
```

No base case, and the return dedented out of the function body. StarCoderData has the lowest held-out loss of any domain (0.70) because code is the most predictable text in the mix; low loss does not mean correct code at this scale.

Usage

Transformers

Requires transformers >= 4.51 (the first release with Qwen3).

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "TevunahAi/Chochmah-1B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).cuda()

prompt = "The Supreme Court of the United States has long held that"
ids = tok(prompt, return_tensors="pt").to("cuda")
out = model.generate(
    **ids,
    max_new_tokens=120,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
    repetition_penalty=1.15,
    no_repeat_ngram_size=4,
)
print(tok.decode(out[0], skip_special_tokens=True))

Prompting a Base Model

• Continue, don't instruct. Give it the opening of the document you want; it will not follow a question or a command.
• The prompt's register selects the corpus. A SCOTUS-style opening produces opinion prose with citations; a textbook sentence produces textbook prose; a def produces code. Write the first line the way the target document would be written.
• No BOS token, no chat template. The tokenizer is saved without one; do not add either.
• Sample tightly. Temperature 0.5–0.7, top-p 0.9, repetition penalty 1.1–1.15, no_repeat_ngram_size=4 (or min_p=0.05), and 100–150 new tokens. Base models do not know when to stop, and most drift lives in the second half of a long generation.

vLLM

pip install vllm
vllm serve TevunahAi/Chochmah-1B --max-model-len 4096 --dtype bfloat16

Fine-Tuning

The model is a stock Qwen3ForCausalLM and works unmodified with TRL, Axolotl, LLaMA-Factory, Unsloth, and PEFT. Its main intended use is as an SFT / DPO base and as a clean, fully documented ~1B target for quantization research.

Known Issues and Limitations

• It will state false things fluently. At this scale the model learns how legal, philosophical, mathematical, and scientific text is written far better than it learns what is true. It must not be used as a source of legal, medical, financial, or factual answers. It is not a substitute for legal advice, and nothing it writes about a case should be assumed to be in that case.
• Real citations, wrong propositions. This is the specific form the previous point takes at 1B, and it is worse than the 350M's because the citations are now usually right. Under greedy decoding the model produced: "the Constitution does not require the States to provide indigent defendants with counsel in all criminal cases. See Gideon v. Wainwright, 372 U.S. 335 , 83 S.Ct. 792 , 9 L.Ed" — a real case, correctly cited, for the opposite of what it held. Treat every citation as a lookup key, never as support.
• Base model. No instruction following, no safety tuning, no chat template, no stop behavior beyond <|endoftext|>.
• Code remains the weakest register. Indentation can break after a docstring and base cases go missing (see the Fibonacci sample); API details are plausible rather than true (a sample placed a derivative() function in scipy.integrate). Expect HumanEval near zero.
• Cosmopedia fingerprints. The synthetic textbook register is the model's default expository voice: references to an "opening extract" that does not exist, and survey-course summaries where a primary-text voice was wanted.
• Formatting artifacts from the data. Citations carry CourtListener's HTML-to-text spacing (467 U.S. 1025, 1031 , 104 S.Ct. 2885 ,). The Gutenberg hard-wrap artifact documented on the 350M is fixed in this model's narrative and philosophy corpora.
• English only, 4,096-token context. Some Wikipedia-derived knowledge is dated 2023-11.
• Repetition at long generation lengths, as with all small models; see the sampling guidance above.

Memory Requirements

Inference

• fp32 (as released): 4.73 GB weights; ~5.5 GB VRAM at full 4,096 context (the GQA KV cache is ~0.4 GB)
• bf16: ~2.4 GB weights; ~3 GB VRAM at full context
• Runs on CPU; ~50 tokens/s on a workstation GPU in fp32 without a KV cache, far more with one

Training (Reproduction)

• Used: RTX 5000 Ada (32 GB) + Dual Xeon Max 9480 (128 GB HBM2e + 256 GB DDR5) for the data pipeline
• Peak VRAM: 24.2 GB at micro-batch FILL x 4,096 with torch.compile
• Tokenized data: ~90 GB (uint16 shards) — fits in page cache on 256 GB of RAM, so the GPU never waits on data
• Raw downloads: several hundred GB, dominated by the 53 GB CourtListener bulk export and FineWeb-Edu

Technical Specifications

Specification	Value
Model Family	TevunahAi Chochmah
Variant	1B (v1.0, base)
Total Parameters	1,182,900,224
Non-Embedding Parameters	1,082,236,928
Layers	24
Hidden Size	2,048
Attention Heads	16
KV Heads	4 (GQA)
Head Dimension	128
QK-Norm	Yes, per head, before RoPE
Intermediate Size	5,632
Activation	SiLU (SwiGLU)
Normalization	RMSNorm, eps 1e-5, pre-norm
Positional Encoding	RoPE, theta 50,000
Biases	None
Embeddings	Tied
Context Length	4,096
Vocab Size	49,152
Tokenizer	SmolLM2
Training Tokens	45.00B
Training Steps	42,915
Global Batch	1,048,576 tokens
Schedule	WSD (FILL warmup / stable / 4,291-step cosine decay)
Final Checkpoint	ckpt_42914.pt
Architecture Class	Qwen3ForCausalLM (transformers >= 4.51)

Roadmap

• Chochmah-1B-Instruct — supervised fine-tuning on an open instruction set
• TevunahAi Ultra-Hybrid GPTQ variant of the instruct model

License

Weights: Apache-2.0.

Training data is a mix of ODC-By (FineWeb-Edu, FineMath, OpenWebMath, peS2o), Apache-2.0 (Cosmopedia v2), CC BY 4.0 (OpenStax), CC BY-SA 4.0 (Wikipedia), permissively licensed code (StarCoderData), and public-domain material (Project Gutenberg; United States judicial opinions, which are edicts of government). No data under non-commercial terms was used. Users are responsible for their own compliance with the licenses of the underlying datasets.

Citation

@misc{chochmah1b_2026,
  title  = {Chochmah-1B: a from-scratch pretrained base model on one GPU},
  author = {TevunahAi},
  year   = {2026},
  url    = {https://huggingface.co/TevunahAi/Chochmah-1B},
  note   = {1.18B parameters, 45B tokens, three-phase WSD curriculum with hand-built case-law, philosophy, and narrative corpora}
}

Acknowledgments

• Hugging Face / SmolLM team for the SmolLM2 tokenizer, FineWeb-Edu, FineMath, and Cosmopedia — and for the SmolLM reports that made the education-weighted recipe a known quantity
• Qwen team for the Qwen3 block design and its transformers implementation, which this model's architecture follows so that it loads without custom code
• Free Law Project for the CourtListener bulk data that made the law corpus possible
• Project Gutenberg and the sedthh/gutenberg_english packaging
• OpenStax for publishing textbook sources under CC BY
• BigCode (StarCoderData), Allen AI (peS2o), Wikimedia
• EleutherAI for lm-evaluation-harness, Proof-Pile-2, and the Pythia suite used as the compute-matched reference

---

Pretrained by TevunahAi AI Model Quantization and On-Premises AI Infrastructure From-scratch models, hybrid-architecture quantization (Mamba, MoE, SSM), and enterprise hardware bring-up

https://huggingface.co/TevunahAi