Chochmah-1B-Diffusion - TevunahAi From-Scratch Base Model, Converted to Block Diffusion

This is the block-diffusion version of Chochmah-1B, our 1.18B-parameter model pretrained from scratch on 45 billion tokens on a single workstation GPU. The base model writes one token at a time, left to right. This one writes eight tokens at a time: each block starts fully masked and is filled in over several passes, in whatever order the model is most sure of, with every position able to see the rest of its block. It was made by converting the finished base model and training it on 3 billion more tokens, on the same RTX 5000 Ada the base was trained on, and every number below comes from that run.

It is released for what it is: a from-scratch language model taken from autoregressive generation to block diffusion on one GPU, with every stage public. It is a base model. It continues text; it was not trained to chat or to follow instructions. It is also not a faster model. It writes coherent text at one token per pass, which costs as many network passes as the autoregressive base, and it breaks down when asked to commit more than one token per pass; the decoding section shows exactly where. The intended uses are the ones that benefit from a small, open, fully documented diffusion language model: studying block-diffusion decoding, sampler and schedule research, and as the starting point for an instruct version.

On the architecture: the network is the base model's, unchanged: the Qwen3 block (GQA, per-head QK-norm, SwiGLU, RMSNorm, tied embeddings). Architecture only: no Qwen weights, data or tokenizer, at any stage. It does not load as Qwen3ForCausalLM, because a causal-LM class cannot express the attention pattern block diffusion needs. The model class and its sampler ship in this repository as one self-contained file and load with trust_remote_code=True.

Model Details

Property Value
Developer TevunahAi
Base Model TevunahAi/Chochmah1B (1,182.9M params, 45B tokens, from scratch)
Model Type Decoder-only transformer, Qwen3-architecture block, block discrete diffusion (BD3-LM)
Conversion Causal mask removed, a mask token added, 3.0B tokens of block-diffusion training
Block Size 8 tokens
Generation Blocks left to right; inside a block, iterative unmasking in order of confidence
Context Length 4,096 tokens inherited from the base; diffusion training used 2,048-token windows
Tokenizer SmolLM2 (49,152 BPE) plus one mask token (id 49152)
Release Precision bf16 safetensors (2.4 GB)
Training Hardware 1x NVIDIA RTX 5000 Ada 32 GB
Training Time 174.1 hours wall-clock (November 6–14, 2026)
Stage Base model (text continuation); see Roadmap for the instruct version
Loads With AutoModelForMaskedLM with trust_remote_code=True; tested on transformers 4.57 and 5.19; no other dependencies
License Apache-2.0

How It Generates

An autoregressive model commits one token per network pass, strictly left to right. This model works a block at a time:

  1. A new block starts as 8 <|mask|> tokens appended to the text so far.
  2. In each pass the model predicts every masked position of the block at once. It sees the whole block, including positions already filled on either side, and all the text before the block.
  3. The prediction it is most confident about is committed. The other positions stay masked for the next pass.
  4. When the block is full it is final, and the next block starts.

Blocks come left to right, but inside a block the order is the model's own choice, and a token can be chosen knowing tokens to its right. Earlier blocks never change, so their keys and values are cached. Training used the block discrete denoising diffusion objective of Arriola et al., 2025 (BD3-LM) through the dLLM trainer.

Usage

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model_id = "TevunahAi/Chochmah-1B-Diffusion"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(model_id, trust_remote_code=True).to(device=device, dtype=dtype).eval()

prompt = "The court held that"
ids = tok(prompt, add_special_tokens=False, return_tensors="pt").input_ids[0].to(device)

new = model.diffusion_generate(ids, max_new_tokens=64)                             # greedy, one token per pass
print(prompt + tok.decode(new))

new = model.diffusion_generate(ids, max_new_tokens=64, temperature=0.7, seed=0)    # sampling
print(prompt + tok.decode(new))

trust_remote_code=True is needed on the tokenizer as well as the model; without it, newer transformers releases stop to ask. diffusion_generate returns only the new tokens and stops after the first block that contains the end-of-text token. No BOS token is used, as with the base model.

Argument Default Meaning
max_new_tokens 128 Upper bound on generated tokens
tokens_per_step 1 Positions committed per pass, 1 to 8. Leave it at 1; see the next section
temperature 0.0 0 is greedy; above 0 samples from softmax(logits / temperature)
order "confidence" Commit the most confident positions first; "random" is the alternative
seed None Makes sampling reproducible

Greedy decoding loops on open-ended prompts, as the samples below show; temperature 0.7 is a better default for prose. model(input_ids, attention_mask=...) returns logits under whatever attention pattern you pass (none means every position sees every position), for anyone writing their own sampler.

Serving

This is custom model code. vLLM, text-generation-inference, llama.cpp / GGUF and anything else that expects a standard causal language model will not run it, and model.generate() and pipeline("text-generation") do not apply. Use diffusion_generate.

Decoding: One Token Per Pass

tokens_per_step sets how many positions of a block are committed in each pass. Four prompts, 64 new tokens each, bf16 on the RTX 5000 Ada:

Tokens per pass Passes per block Speed Result
1, greedy 8 ~55 tok/s Coherent text
1, temperature 0.7 8 ~52 tok/s Coherent text, more varied
2, greedy 4 ~100 tok/s One prompt mostly holds; the others drop words and collapse into repetition
4, temperature 0.7 2 ~165 tok/s Unusable

Measured with dLLM's sampler. The sampler bundled in this repository gives the same greedy output token for token; it has not been timed on a GPU.

How to read it. Tokens committed in the same pass are chosen independently of each other. With a whole block still blank, the safest independent guesses are the commonest words, so at four per pass the output fills with "and", "the" and "=". At one per pass every choice sees the ones before it, and the text is coherent. The cliff is at two.

That makes this an honest statement of where a 3B-token conversion of a 1B model lands: block diffusion that works, and no speed advantage. At one token per pass it runs at about the speed of the autoregressive base. The interesting property is the generation order, not the throughput, and the samples at two and four tokens per pass are included below because the failure is as informative as the success.

Training

Data

No new data. The model re-read the first 3.0B tokens of the 4.5B-token final phase of the base model's own pretraining, in 2,048-token windows: synthetic textbooks and stories (Cosmopedia) 36%, FineMath 17%, StarCoder code 12%, philosophy 10%, canonical case law 10%, FineWeb-Edu 10%, narrative fiction 4%, and under 1% each of OpenWebMath and textbooks. Sources and licenses are on the Chochmah-1B card.

Using data the base had already seen is deliberate. The conversion has to teach a new way of generating, not new knowledge, and familiar text isolates the one thing that changed. Everything this model knows, and everything it gets wrong, is the base model's.

Sixty-four windows were held out and scored every 100 steps.

Conversion

The base checkpoint was loaded into a model class with no built-in causal mask, and one token was added: <|mask|>, id 49152, with the embedding matrix grown from 49,152 to 49,216 rows and the new rows initialised to the mean of the old ones. Before any diffusion training, the converted model run with a causal mask reproduced the base model exactly (argmax agreement 1.0000, cross-entropy 1.455 on final-phase windows), so the run starts from the released base model and not from an approximation of it.

Recipe

Field Value
Method Full-parameter continued pretraining with the BD3-LM objective (no LoRA, no frozen layers)
Objective Per sequence, a masking rate t drawn uniformly from (0.001, 1); every token masked with probability t; cross-entropy on the masked tokens, weighted by 1/t
Block Size 8
Attention A noised block sees itself and the clean text before it; the clean text is block-causal. Built as a block-sparse PyTorch flex-attention mask over the noised and clean copies together (4,096 positions)
Sequence Length 2,048 tokens
Optimizer 8-bit AdamW (bitsandbytes), betas (0.9, 0.999), eps 1e-8, weight decay 0.1
Learning Rate 1e-4 peak, 200-step linear warmup, cosine decay to 0
Gradient Clipping 1.0
Global Batch 524,288 tokens (1 sequence x 256 accumulation steps x 2,048)
Steps / Tokens 5,722 / 3.0B
Precision fp32 master weights, bf16 autocast (the base model's scheme)
Memory No gradient checkpointing; peak 22.3 GB
Throughput ~4,800 tok/s
Framework dLLM BD3LMTrainer (commit ca17675), transformers 4.57, torch.compile

The attention implementation is what made the run fit in a week. With a dense 4,096 x 4,096 mask and gradient checkpointing the same step ran at 2,100 tok/s on this card; with flex attention, no checkpointing and torch.compile it ran at 5,100.

One correction on top of dLLM for anyone reproducing this with gradient accumulation: its trainer normalises the loss per micro-batch, and with transformers 4.57 the division by the accumulation steps is then skipped, which multiplies the effective learning rate by the accumulation count. Setting model_accepts_loss_kwargs = False on the trainer restores it.

Loss

The converted model starts at a loss of about 8.3 under the diffusion objective: it has never seen a mask token. It is near 2.2 by step 1,400 and 2.08 over the final steps; the average over the whole run is 2.19.

These are values of the diffusion objective, a variational upper bound on the negative log-likelihood, and are not comparable with the base model's cross-entropy of 1.455 on this data.

The held-out number is noisy and is reported as such. Each evaluation scores the 64 held-out windows under freshly drawn masks, and single readings swing by a couple of tenths with no change in the model: 1.94 at step 5,400 and 1.71 at step 5,700, both with the learning rate already below 1e-6. Neither is the number; the training loss, which averages 2,560 sequences per logged value, is the steadier measure.

Evaluation

Not benchmarked yet. The base model's scorecard (ARC-Easy, HellaSwag, PIQA, LAMBADA against compute-matched references) is on the Chochmah-1B card. The conversion added no data, so those numbers are the reference this model will be measured against, and the same tasks run through the diffusion model will be added here. Until then the evidence for quality is the samples below, which is four prompts and should be weighed as four prompts.

Sample Outputs

Verbatim, unedited, from the first prompts run on the finished weights (bf16, RTX 5000 Ada, one token per pass, greedy, 64 new tokens, so each stops mid-sentence). Shown together on purpose: the same model, four prompts, and between them an accurate picture of what it is.

Prompt: The court held that

the use of the word was not a part of the contract, and that the plaintiff was not entitled to recover.
 
 
 The opinion of the court in
 
 Pullman
 
 v.
 
 Marshall,
 
 179 U. S. 360 , is applicable to the present

The register of an old contract opinion, a holding, then a citation in correct reporter form. The line breaks around the case name are in the output; the model reproduces a layout it evidently learned from its case-law data. Run the same prompt in float32 on a CPU and the text is identical up to the case name, which comes out as Smith v. Smith, 105 U. S. 150. The model is not recalling a case: the same weights on different hardware cite a different one. Treat any citation it produces as a lookup key, never as support.

Prompt: In the beginning

, the whole of the world was a mass of water, and the inhabitants of the earth were all fishes. But one day the waters of the world began to rise, and the waters of the earth began to fall, and the inhabitants of the earth began to rise, and the inhabitants of the sea began to fall,

An invented cosmogony in a folk-tale voice, fluent for two sentences, then a rise-and-fall loop: greedy decoding with nothing to break the pattern.

Prompt: def quicksort(arr):

# Sort the array
arr.sort()

# Initialize the pivot element
p = 0

# Count the number of elements to the left of the pivot
left = 0

# Count the number of elements to the right of the pivot
right = 0

while

Commented, plausible at a glance, and not quicksort: it calls arr.sort() and then starts setting up counters. The body is flush left. Indentation is a weakness the base model already had (the instruct card shows it dedenting a return); here the whole body lost it.

Prompt: The categorical imperative, according to Kant,

is the fundamental principle of all rational knowledge. It is the principle that we can only know what we can will. It is the principle that we can only will what we can know. It is the principle that we can only know what we can will. It is the principle that we can only will what we can know

The first clause has the sound of a definition and the wrong content: the categorical imperative is Kant's supreme principle of morality, not of knowledge. The know-and-will sentence that follows is not Kant's either, and then it loops.

More samples β€” the same prompts sampled at temperature 0.7

Prompt: The court held that

the prosecutor did not violate the principles of the Due Process Clause by failing to advise the jury of the consequences of the felony-murder rule. The court noted that the prosecutor did not consult the jury on the consequences of the felony-murder rule. The court held that the prosecutor did not violate the Due

Real doctrines, correctly named, arranged into a holding that does not make legal sense: instructing the jury is the judge's job, not the prosecutor's.

Prompt: In the beginning

, there was a group of friends who loved playing soccer and hiking together. They would spend hours on their feet, exploring new trails and seeing beautiful sights. But one day, they realized that keeping up with their activities and staying fit was becoming quite difficult.

That's when they turned to technology for help. They

The synthetic-story register of the training mix. Sampling removes the loop that greedy decoding fell into.

Prompt: The categorical imperative, according to Kant,

is the principle of the categorical judgment.

The categorical imperative is the principle of the categorical judgment.

It is the principle of the categorical judgment that makes the imperative imperative. It is the principle of the categorical judgment that makes man a rational.

The categorical imperative is the principle of the categorical judgment.

Sampling does not rescue this one. On Kant the model has the vocabulary and none of the argument.

What more than one token per pass looks like

Two tokens per pass, greedy. Prompt: The categorical imperative, according to Kant, (the last line then repeats to the end):

is the foundation of of the moral of the. .

The. categorical of the.

The. categorical of the.

Four tokens per pass, temperature 0.7. Prompt: In the beginning

of there's book, according to the Scriptures,, Abraham, there, and his, were three, and, and and and and and and and, and, and and and and and and and and and and and and and where and, and and and and and and and and and and and and and and and

Prompt: def quicksort(arr):

# n = = . =  = = = n = = = = = = = = = = = = = = …

The code sample goes on in = signs to the end. The words that pile up are the ones that are safest to guess with no context: the commonest token of each register.

Known Issues and Limitations

  • The knowledge is the base model's. The conversion re-read data the base had already seen and added none. This model gives fluent, confident, wrong accounts of specific cases, texts and ideas, as the samples show. It must not be used as a source of legal, medical, financial or factual answers.
  • Citations are generated, not retrieved. A case name and reporter citation come out the same way as any other words, and the same prompt can cite a different case on different hardware. Treat any citation as a lookup key, never as support.
  • A base model, not an assistant. No chat format, no instruction following. It continues the text it is given.
  • One token per pass. Two per pass already breaks most outputs and four is unusable. There is no speed advantage over the autoregressive base.
  • Greedy decoding loops on open-ended prompts. Temperature around 0.7 helps prose; it does not help where the model has nothing to say.
  • Code comes out unindented, and is not reliable code.
  • Not benchmarked yet. The only quality evidence so far is a handful of samples.
  • No safety tuning. It has no refusal training and no content filtering of its own.
  • English only. Diffusion training used 2,048-token windows; behaviour beyond that length is untested.
  • Custom code. It runs through transformers with trust_remote_code=True and nowhere else: no vLLM, no GGUF, no generate().

Technical Specifications

Specification Value
Model Family TevunahAi Chochmah
Variant 1B-Diffusion (v1.0, base)
Base Chochmah-1B, final checkpoint ckpt_42914
Total Parameters 1,183,031,296 (the base's 1,182,900,224 plus 64 embedding rows)
Architecture Class ChochmahDiffusionLM, in modeling_chochmah_diffusion.py in this repository
Layers / Hidden / Intermediate 24 / 2,048 / 5,632
Attention GQA, 16 query heads, 4 KV heads, head dim 128, QK-norm; no built-in causal mask
Positions RoPE, theta 50,000
Context Length 4,096 (diffusion-trained at 2,048)
Vocab Size 49,153 tokens (SmolLM2's 49,152 plus the mask token); 49,216 embedding rows
Objective BD3-LM block diffusion, block size 8
Diffusion Training 3.0B tokens, 5,722 steps
Compute 174.1 h on one RTX 5000 Ada
Final Training Loss 2.08 (diffusion objective)

Roadmap

  • Chochmah-1B-Diffusion-Instruct β€” block-diffusion fine-tuning on SmolTalk from these weights, the diffusion counterpart of Chochmah-1B-Instruct
  • Benchmarks β€” the base model's scorecard tasks run through this model, to measure what the conversion kept

License

Weights and model code: Apache-2.0. The base model's pretraining data licenses are listed on the Chochmah-1B card. Users are responsible for their own compliance with the licenses of the underlying datasets.

Citation

@misc{chochmah1b_diffusion_2026,
  title  = {Chochmah-1B-Diffusion: converting a from-scratch language model to block diffusion on one GPU},
  author = {TevunahAi},
  year   = {2026},
  url    = {https://huggingface.co/TevunahAi/Chochmah-1B-Diffusion},
  note   = {BD3-LM conversion of Chochmah-1B; 1.18B parameters, block size 8}
}

Acknowledgments

  • Marianne Arriola and co-authors for block discrete diffusion (BD3-LM), the objective this model was trained with
  • The dLLM authors for an open trainer and the autoregressive-to-diffusion recipe this conversion follows
  • Qwen team for the Qwen3 block design and its transformers implementation
  • Hugging Face / SmolLM team for the tokenizer
  • PyTorch team for flex attention, which made block-sparse attention over 4,096 positions practical on one GPU

Pretrained and converted by TevunahAi
AI Model Quantization and On-Premises AI Infrastructure
From-scratch models, hybrid-architecture quantization (Mamba, MoE, SSM), and enterprise hardware bring-up

https://huggingface.co/TevunahAi https://tevunah.ai https://chochmah.ai

Downloads last month
63
Safetensors
Model size
1B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for TevunahAi/Chochmah-1B-Diffusion

Finetuned
(1)
this model

Paper for TevunahAi/Chochmah-1B-Diffusion