README.md

---
license: apache-2.0
language:
- en
tags:
- language-model
- transformer
- rope
- swiglu
- diffusion
- masked-diffusion
- discrete-diffusion
- from-scratch
- tiny
- small
- experimental
datasets:
- HuggingFaceFW/fineweb-edu
- mlfoundations/dclm-baseline-1.0
- HuggingFaceTB/smollm-corpus
- HuggingFaceTB/finemath
- SimpleStories/SimpleStories
pipeline_tag: text-generation
library_name: transformers
---

# ExpIvme-DiffusionConversate-v1

**İvme** (Turkish: *acceleration*) is normally a series of small autoregressive language models. This is not one of those. This is the same architecture family run through a different objective entirely: a masked/absorbing-state discrete diffusion language model instead of next-token prediction. "Exp" is doing real work in the name — this is an experiment to see what a small dSLM (diffusion small language model) can do at a scale we can train on one GPU in an afternoon, not a production model.

Short version: the pipeline works, the sampler had a real bug that we found and fixed, and the model is too small and too lightly trained to have learned much beyond local fluency. All three of those are reported honestly below.

---

## Model Details

| Parameter | Value |
|---|---|
| Architecture | Bidirectional transformer, dense, masked diffusion (not causal) |
| Parameters | 130.1M (unique; tied embeddings counted once) |
| Layers | 12 |
| Hidden dim | 896 |
| FFN | SwiGLU |
| Attention heads | 14, full bidirectional attention (no causal mask) |
| Context length | 1024 tokens |
| Vocab size | 16,001 (16,000 BPE + 1 mask token) |
| Positional encoding | RoPE (θ=10,000), real-valued cos/sin formulation |
| Normalization | RMSNorm (pre-norm) |
| Embeddings | Tied input/output |
| Biases | None |
| Diffusion process | Absorbing-state (masking), per-example mask rate t ~ U(0.001, 1.0) |

Architecture lineage: [IvmeLabs/Ivme-Conversate-v2-Base](https://huggingface.co/IvmeLabs/Ivme-Conversate-v2-Base), same RMSNorm/RoPE/SwiGLU/tied-embedding building blocks, scaled from 24M to 130M params, with the one structural change masked diffusion actually requires: bidirectional attention instead of causal, since the model needs to see both sides of a masked token to denoise it.

<a href="https://hfviewer.com/IvmeLabs/ExpIvme-DiffusionConversate-v1?utm_source=huggingface&amp;utm_medium=embedded_model_card&amp;utm_campaign=IvmeLabs__ExpIvme-DiffusionConversate-v1_card&amp;utm_content=embedded_card_open_viewer&amp;from=embedded-model-card" target="_blank" rel="noopener">
  <img
    src="https://hfviewer.com/api/card.svg?source=IvmeLabs%2FExpIvme-DiffusionConversate-v1&amp;granularity=auto&amp;v=20260516-title-pills-card"
    alt="Architecture graph for IvmeLabs/ExpIvme-DiffusionConversate-v1. Open in hfviewer"
    width="100%"
  />
</a>

---

## Why diffusion, and what that changes

Standard causal LMs predict the next token given everything before it. This model instead learns to fill in an arbitrary subset of masked positions given everything around them, at a training-time mask rate sampled uniformly per example. At generation time, you start from a fully masked sequence and iteratively unmask positions over a fixed number of steps, rather than emitting one token at a time left to right.

That has real consequences for anything downstream:

- **No single forward pass gives you a likelihood.** Training loss is a variational bound (ELBO), not exact log-likelihood, so anything that needs "the model's probability of this text" (perplexity, multiple-choice scoring) needs a diffusion-appropriate substitute, not the AR formula. See Evaluation below for how we handled this.
- **Sampling has its own failure modes AR decoding doesn't.** We hit one during this project: see Known Issues.
- **Step count is a real, tunable quality/cost knob** in a way it just isn't for AR models. More steps costs more compute and (up to a point) buys better generations.

---

## Training

### Data mix (~1.75B tokens)

| Source | Share | Tokens |
|---|---|---|
| HuggingFaceFW/fineweb-edu | 50% | 875M |
| mlfoundations/dclm-baseline-1.0 | 25% | 437.5M |
| HuggingFaceTB/smollm-corpus (cosmopedia-v2) | 12% | 210M |
| HuggingFaceTB/finemath (finemath-3plus) | 8% | 140M |
| SimpleStories/SimpleStories | 5% | 87.5M |

For reference, Ivme-Conversate-v2-Base (the AR sibling this architecture is descended from) trained on 12.85B tokens, roughly 7x more than this run. That gap matters for interpreting the results below — this is not a controlled diffusion-vs-AR comparison, since neither the token budget nor the objective is held constant. It's one data point on what this specific budget gets you with this specific objective.

### Hyperparameters

| Setting | Value |
|---|---|
| Optimizer | AdamW |
| Peak LR | 3e-4 |
| LR schedule | Cosine, 1000-step warmup, decays to 3e-5 |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Batch size | 128 sequences x 1024 tokens (131,072 tokens/step), auto-probed against GPU memory at start of training |
| Total steps | 13,351 |
| Precision | bfloat16 |
| Attention | PyTorch `scaled_dot_product_attention`, cuDNN backend explicitly pinned (the fast path on sm_120; auto-dispatch is not reliable for masked/bidirectional attention on this hardware) |
| Compilation | `torch.compile`; RoPE was rewritten from a complex-tensor formulation to real-valued cos/sin specifically because Inductor cannot generate fused code for complex ops and was silently falling back to eager execution for it |
| Loss | Masked cross-entropy at masked positions, weighted by 1/t per example (standard NELBO reweighting for absorbing-state diffusion) |

### Hardware

Trained on a single NVIDIA RTX PRO 6000 Blackwell Server Edition (96GB, sm_120) in approximately **2 hours**, sustaining roughly 215–245k tokens/sec.

### Loss curve

Per-step loss in masked diffusion training is inherently noisier than AR training — the 1/t reweighting means a single unlucky low-t batch can be weighted 10–100x more heavily than a typical one, purely from how t happens to be sampled that step, independent of how well the model is doing. A 10-step moving average over the run shows the real trend: mean loss dropped from roughly 65 in the first few hundred steps to roughly 33 by the final few hundred, with the noise (standard deviation) shrinking by a similar factor as training progressed. Individual step-to-step loss values are not a meaningful signal on their own for this objective; the smoothed trend is.

---

## Evaluation

### Method

Neither perplexity nor multiple-choice log-likelihood ranking transfers directly from AR evaluation to this model, since there's no single causal forward pass to score against. Following the convention used across the masked-diffusion literature (SEDD, MDLM, and related work all report the same kind of substitute), we use **ELBO-based scoring**: mask only the span being evaluated, keep everything else as fixed context, and compute the same 1/t-weighted masked loss used in training, averaged over a fixed grid of t values rather than one noisy sample. Lower loss means the model prefers those tokens over the alternative.

### ARC-Easy

| Metric | Result |
|---|---|
| Accuracy (ELBO-ranked, 4-way) | 25.51% (606 / 2,376) |
| Random baseline | 25.00% |

This is not a meaningful result above chance. We checked for scoring artifacts before concluding this — predicted-answer letter distribution matches the true-answer distribution almost exactly (no position bias), and the gap between the best- and second-best-scoring option is small and uncorrelated with whether the prediction was actually correct, in both cases consistent with the model genuinely having no discriminative signal on this task rather than a broken scorer. At 130M parameters and 1.75B tokens, this model has not absorbed enough factual/scientific content for ARC-Easy-style recall to show up, which is not unusual for similarly-sized AR models at comparable token budgets either — this looks like a scale finding more than an architecture finding, though we did not run a controlled AR baseline to confirm that directly.

We have not yet run a WikiText-2 ELBO/perplexity-style evaluation or a generation-quality pass on this checkpoint; both are natural next steps if this line of work continues.

---

## Known Issues

**Confidence-based unmasking collapses into repetition without noise perturbation.** The standard decoding strategy for masked diffusion unmasks the highest-confidence positions first each step. On this model, that created a feedback loop: once a few high-frequency tokens (periods, in practice) got placed, the model became increasingly confident about placing more of the same nearby, and confidence-ranked selection kept picking exactly those positions, compounding over steps. At 32+ sampling steps this reliably collapsed 40–55% of the output into a single repeated token.

Fix: rank by `log(confidence) + gumbel_temp * Gumbel(0,1) noise` instead of raw confidence (the standard LLaDA/MaskGIT-style mitigation), which at `gumbel_temp=0` is mathematically identical to the broken behavior and at `gumbel_temp≈1.0` breaks the collapse without degrading into pure noise. We recommend `gumbel_temp=1.0` as a starting point for anyone sampling from this checkpoint; a small sweep (`gumbel_temp` in `{0.6, 1.0, 1.5}`) is cheap and worth doing per use case, since we found the curve non-monotonic — too much noise introduces a different collapse (onto whatever token happens to benefit most from the perturbation, not the model's actual preference).

We did not have this fixed at initial release; the sample generation in earlier internal testing used plain confidence-based unmasking and looked considerably worse than the model's actual learned distribution supports. If you're comparing this model against older cached outputs, re-generate with noise-perturbed unmasking first.

---

## Inference

```python
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained(
    "IvmeLabs/ExpIvme-DiffusionConversate-v1", trust_remote_code=True,
).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained(
    "IvmeLabs/ExpIvme-DiffusionConversate-v1", trust_remote_code=True,
)
mask_token_id = model.config.mask_token_id

@torch.no_grad()
def sample(length=96, steps=32, temperature=1.0, gumbel_temp=1.0):
    input_ids = torch.full((1, length), mask_token_id, dtype=torch.long, device="cuda")
    for step in range(steps):
        logits = model(input_ids=input_ids).logits
        probs = F.softmax(logits / temperature, dim=-1)
        sampled = torch.multinomial(probs.view(-1, probs.size(-1)), 1).view(1, length)

        is_masked = input_ids == mask_token_id
        n_masked = is_masked.sum().item()
        if n_masked == 0:
            break

        frac_remaining = 1.0 - (step + 1) / steps
        denom = max(1 - step / steps, 1e-6)
        n_to_unmask = min(max(1, int(n_masked * (1 - frac_remaining / denom))), n_masked)

        conf = probs.gather(-1, sampled.unsqueeze(-1)).squeeze(-1)
        log_conf = torch.log(conf.clamp(min=1e-9))
        u = torch.rand_like(conf).clamp(min=1e-9, max=1 - 1e-9)
        gumbel_noise = -torch.log(-torch.log(u))
        score = (log_conf + gumbel_temp * gumbel_noise).masked_fill(~is_masked, float("-inf"))

        topk = torch.topk(score, k=n_to_unmask, dim=-1).indices
        update_mask = torch.zeros_like(is_masked).scatter_(1, topk, True)
        input_ids = torch.where(update_mask, sampled, input_ids)

    return tokenizer.decode(input_ids[0].tolist())

print(sample())
```

`trust_remote_code=True` is required (custom architecture: bidirectional RoPE + SwiGLU + RMSNorm masked diffusion transformer). There is no `.generate()` support — diffusion sampling isn't next-token generation, so the sampler above is the actual inference path, not a convenience wrapper around something else.

---

## Limitations

- Experimental. Not instruction tuned. No chat template, no conversational behavior.
- At-chance performance on ARC-Easy (see Evaluation); we would not expect meaningful factual recall on other knowledge-heavy benchmarks either at this scale/token budget.
- English only.
- 1024 token context window.
- No AR baseline was trained at matched size/data/compute, so nothing above should be read as a controlled diffusion-vs-autoregressive comparison — it's a standalone characterization of this run.
- Default confidence-based sampling is broken without the noise-perturbation fix described in Known Issues; use the inference code above, not a naive top-confidence sampler.
- 1.75B training tokens is well below typical AR pretraining budgets for a 130M-parameter model; we don't know what a compute-matched or token-matched run would show, since we didn't run one.

---

## What's Next

If this line of work continues: a WikiText-2 ELBO evaluation, a generation-quality pass with the corrected sampler, a steps-vs-quality sweep (a genuinely diffusion-specific axis with no AR equivalent), and — if the result above is worth chasing further — a proper token-matched or compute-matched AR baseline trained on the identical architecture and data mix, so any future diffusion-vs-AR claim would actually be a controlled comparison instead of two numbers from different runs.

You can check our other models on our organization card.

---

## Citation

```bibtex
@misc{expivme-diffusionconversate-v1,
  author       = {IvmeLabs},
  title        = {ExpIvme-DiffusionConversate-v1},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/IvmeLabs/ExpIvme-DiffusionConversate-v1}
}
```

---

*Built by IvmeLabs. Small models, deliberate choices, and this time, an honest null result.*