ExpIvme-DiffusionConversate-v1

İvme (Turkish: acceleration) is normally a series of small autoregressive language models. This is not one of those. This is the same architecture family run through a different objective entirely: a masked/absorbing-state discrete diffusion language model instead of next-token prediction. "Exp" is doing real work in the name — this is an experiment to see what a small dSLM (diffusion small language model) can do at a scale we can train on one GPU in an afternoon, not a production model.

Short version: the pipeline works, the sampler had a real bug that we found and fixed, and the model is too small and too lightly trained to have learned much beyond local fluency. All three of those are reported honestly below.

---

Model Details

Parameter	Value
Architecture	Bidirectional transformer, dense, masked diffusion (not causal)
Parameters	130.1M (unique; tied embeddings counted once)
Layers	12
Hidden dim	896
FFN	SwiGLU
Attention heads	14, full bidirectional attention (no causal mask)
Context length	1024 tokens
Vocab size	16,001 (16,000 BPE + 1 mask token)
Positional encoding	RoPE (θ=10,000), real-valued cos/sin formulation
Normalization	RMSNorm (pre-norm)
Embeddings	Tied input/output
Biases	None
Diffusion process	Absorbing-state (masking), per-example mask rate t ~ U(0.001, 1.0)

Architecture lineage: IvmeLabs/Ivme-Conversate-v2-Base, same RMSNorm/RoPE/SwiGLU/tied-embedding building blocks, scaled from 24M to 130M params, with the one structural change masked diffusion actually requires: bidirectional attention instead of causal, since the model needs to see both sides of a masked token to denoise it.

---

Why diffusion, and what that changes

Standard causal LMs predict the next token given everything before it. This model instead learns to fill in an arbitrary subset of masked positions given everything around them, at a training-time mask rate sampled uniformly per example. At generation time, you start from a fully masked sequence and iteratively unmask positions over a fixed number of steps, rather than emitting one token at a time left to right.

That has real consequences for anything downstream:

• No single forward pass gives you a likelihood. Training loss is a variational bound (ELBO), not exact log-likelihood, so anything that needs "the model's probability of this text" (perplexity, multiple-choice scoring) needs a diffusion-appropriate substitute, not the AR formula. See Evaluation below for how we handled this.
• Sampling has its own failure modes AR decoding doesn't. We hit one during this project: see Known Issues.
• Step count is a real, tunable quality/cost knob in a way it just isn't for AR models. More steps costs more compute and (up to a point) buys better generations.

---

Training

Data mix (~1.75B tokens)

Source	Share	Tokens
HuggingFaceFW/fineweb-edu	50%	875M
mlfoundations/dclm-baseline-1.0	25%	437.5M
HuggingFaceTB/smollm-corpus (cosmopedia-v2)	12%	210M
HuggingFaceTB/finemath (finemath-3plus)	8%	140M
SimpleStories/SimpleStories	5%	87.5M

For reference, Ivme-Conversate-v2-Base (the AR sibling this architecture is descended from) trained on 12.85B tokens, roughly 7x more than this run. That gap matters for interpreting the results below — this is not a controlled diffusion-vs-AR comparison, since neither the token budget nor the objective is held constant. It's one data point on what this specific budget gets you with this specific objective.

Hyperparameters

Setting	Value
Optimizer	AdamW
Peak LR	3e-4
LR schedule	Cosine, 1000-step warmup, decays to 3e-5
Weight decay	0.1
Gradient clipping	1.0
Batch size	128 sequences x 1024 tokens (131,072 tokens/step), auto-probed against GPU memory at start of training
Total steps	13,351
Precision	bfloat16
Attention	PyTorch scaled_dot_product_attention, cuDNN backend explicitly pinned (the fast path on sm_120; auto-dispatch is not reliable for masked/bidirectional attention on this hardware)
Compilation	torch.compile; RoPE was rewritten from a complex-tensor formulation to real-valued cos/sin specifically because Inductor cannot generate fused code for complex ops and was silently falling back to eager execution for it
Loss	Masked cross-entropy at masked positions, weighted by 1/t per example (standard NELBO reweighting for absorbing-state diffusion)

Hardware

Trained on a single NVIDIA RTX PRO 6000 Blackwell Server Edition (96GB, sm_120) in approximately 2 hours, sustaining roughly 215–245k tokens/sec.

Loss curve

Per-step loss in masked diffusion training is inherently noisier than AR training — the 1/t reweighting means a single unlucky low-t batch can be weighted 10–100x more heavily than a typical one, purely from how t happens to be sampled that step, independent of how well the model is doing. A 10-step moving average over the run shows the real trend: mean loss dropped from roughly 65 in the first few hundred steps to roughly 33 by the final few hundred, with the noise (standard deviation) shrinking by a similar factor as training progressed. Individual step-to-step loss values are not a meaningful signal on their own for this objective; the smoothed trend is.

---

Evaluation

Method

Neither perplexity nor multiple-choice log-likelihood ranking transfers directly from AR evaluation to this model, since there's no single causal forward pass to score against. Following the convention used across the masked-diffusion literature (SEDD, MDLM, and related work all report the same kind of substitute), we use ELBO-based scoring: mask only the span being evaluated, keep everything else as fixed context, and compute the same 1/t-weighted masked loss used in training, averaged over a fixed grid of t values rather than one noisy sample. Lower loss means the model prefers those tokens over the alternative.

ARC-Easy

Metric	Result
Accuracy (ELBO-ranked, 4-way)	25.51% (606 / 2,376)
Random baseline	25.00%

This is not a meaningful result above chance. We checked for scoring artifacts before concluding this — predicted-answer letter distribution matches the true-answer distribution almost exactly (no position bias), and the gap between the best- and second-best-scoring option is small and uncorrelated with whether the prediction was actually correct, in both cases consistent with the model genuinely having no discriminative signal on this task rather than a broken scorer. At 130M parameters and 1.75B tokens, this model has not absorbed enough factual/scientific content for ARC-Easy-style recall to show up, which is not unusual for similarly-sized AR models at comparable token budgets either — this looks like a scale finding more than an architecture finding, though we did not run a controlled AR baseline to confirm that directly.

We have not yet run a WikiText-2 ELBO/perplexity-style evaluation or a generation-quality pass on this checkpoint; both are natural next steps if this line of work continues.

---

Known Issues

Confidence-based unmasking collapses into repetition without noise perturbation. The standard decoding strategy for masked diffusion unmasks the highest-confidence positions first each step. On this model, that created a feedback loop: once a few high-frequency tokens (periods, in practice) got placed, the model became increasingly confident about placing more of the same nearby, and confidence-ranked selection kept picking exactly those positions, compounding over steps. At 32+ sampling steps this reliably collapsed 40–55% of the output into a single repeated token.

Fix: rank by log(confidence) + gumbel_temp * Gumbel(0,1) noise instead of raw confidence (the standard LLaDA/MaskGIT-style mitigation), which at gumbel_temp=0 is mathematically identical to the broken behavior and at gumbel_temp≈1.0 breaks the collapse without degrading into pure noise. We recommend gumbel_temp=1.0 as a starting point for anyone sampling from this checkpoint; a small sweep (gumbel_temp in {0.6, 1.0, 1.5}) is cheap and worth doing per use case, since we found the curve non-monotonic — too much noise introduces a different collapse (onto whatever token happens to benefit most from the perturbation, not the model's actual preference).

We did not have this fixed at initial release; the sample generation in earlier internal testing used plain confidence-based unmasking and looked considerably worse than the model's actual learned distribution supports. If you're comparing this model against older cached outputs, re-generate with noise-perturbed unmasking first.

---

Inference

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained(
    "IvmeLabs/ExpIvme-DiffusionConversate-v1", trust_remote_code=True,
).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained(
    "IvmeLabs/ExpIvme-DiffusionConversate-v1", trust_remote_code=True,
)
mask_token_id = model.config.mask_token_id

@torch.no_grad()
def sample(length=96, steps=32, temperature=1.0, gumbel_temp=1.0):
    input_ids = torch.full((1, length), mask_token_id, dtype=torch.long, device="cuda")
    for step in range(steps):
        logits = model(input_ids=input_ids).logits
        probs = F.softmax(logits / temperature, dim=-1)
        sampled = torch.multinomial(probs.view(-1, probs.size(-1)), 1).view(1, length)

        is_masked = input_ids == mask_token_id
        n_masked = is_masked.sum().item()
        if n_masked == 0:
            break

        frac_remaining = 1.0 - (step + 1) / steps
        denom = max(1 - step / steps, 1e-6)
        n_to_unmask = min(max(1, int(n_masked * (1 - frac_remaining / denom))), n_masked)

        conf = probs.gather(-1, sampled.unsqueeze(-1)).squeeze(-1)
        log_conf = torch.log(conf.clamp(min=1e-9))
        u = torch.rand_like(conf).clamp(min=1e-9, max=1 - 1e-9)
        gumbel_noise = -torch.log(-torch.log(u))
        score = (log_conf + gumbel_temp * gumbel_noise).masked_fill(~is_masked, float("-inf"))

        topk = torch.topk(score, k=n_to_unmask, dim=-1).indices
        update_mask = torch.zeros_like(is_masked).scatter_(1, topk, True)
        input_ids = torch.where(update_mask, sampled, input_ids)

    return tokenizer.decode(input_ids[0].tolist())

print(sample())

trust_remote_code=True is required (custom architecture: bidirectional RoPE + SwiGLU + RMSNorm masked diffusion transformer). There is no .generate() support — diffusion sampling isn't next-token generation, so the sampler above is the actual inference path, not a convenience wrapper around something else.

---

Limitations

• Experimental. Not instruction tuned. No chat template, no conversational behavior.
• At-chance performance on ARC-Easy (see Evaluation); we would not expect meaningful factual recall on other knowledge-heavy benchmarks either at this scale/token budget.
• English only.
• 1024 token context window.
• No AR baseline was trained at matched size/data/compute, so nothing above should be read as a controlled diffusion-vs-autoregressive comparison — it's a standalone characterization of this run.
• Default confidence-based sampling is broken without the noise-perturbation fix described in Known Issues; use the inference code above, not a naive top-confidence sampler.
• 1.75B training tokens is well below typical AR pretraining budgets for a 130M-parameter model; we don't know what a compute-matched or token-matched run would show, since we didn't run one.

---

What's Next

If this line of work continues: a WikiText-2 ELBO evaluation, a generation-quality pass with the corrected sampler, a steps-vs-quality sweep (a genuinely diffusion-specific axis with no AR equivalent), and — if the result above is worth chasing further — a proper token-matched or compute-matched AR baseline trained on the identical architecture and data mix, so any future diffusion-vs-AR claim would actually be a controlled comparison instead of two numbers from different runs.

You can check our other models on our organization card.

---

Citation

@misc{expivme-diffusionconversate-v1,
  author       = {IvmeLabs},
  title        = {ExpIvme-DiffusionConversate-v1},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/IvmeLabs/ExpIvme-DiffusionConversate-v1}
}

---

Built by IvmeLabs. Small models, deliberate choices, and this time, an honest null result.