metadata
language: en
tags:
- mask-predict
- diffusion
- masked-lm
library_name: transformers
base_model: answerdotai/ModernBERT-base
pipeline_tag: fill-mask
modernbert-diffusion-universal
Model Summary
A diffusion-style masked language model fine-tuned in universal mode using a discrete denoising objective.
Model Details
- Model ID: philipp-zettl/modernbert-diffusion-universal
- Base model: answerdotai/ModernBERT-base
- Training mode: universal
- Task type: Masked token denoising / diffusion-style infilling
Intended Use
Intended as a general-purpose infilling model across text, code, JSON, and chat formats.
Example
from refinebert.diffusion_engine import MaskedDiffusionEngine
engine = MaskedDiffusionEngine("philipp-zettl/modernbert-diffusion-universal")
prompt = "def generate_json(data):"
output = engine.generate(prompt, num_new_tokens=25, steps=12, guidance_scale=3.0)
print(output)
Training Data
Datasets are streamed from Hugging Face and mixed by mode.
Dataset Mix
| Dataset | Percentage | Purpose |
|---|---|---|
| HuggingFaceFW/fineweb-edu (sample-10BT) | 40% | General web/edu text |
| bigcode/the-stack-dedup (python) | 30% | Python code |
| bigcode/the-stack-dedup (json) | 15% | Structured JSON |
| HuggingFaceH4/ultrachat_200k (train_sft) | 15% | Instruction chat |
Fallbacks: FineWeb-Edu may fall back to Wikitext-103, and The Stack may fall back to CodeParrot depending on availability.
Training Procedure
- Steps: 500000
- Batch size: 16
- Sequence length: 256
- Learning rate: 5e-05
- CFG dropout probability: 0.1
- Samples loaded into RAM: 100000
Training Time & Hardware
- Duration: 46h 38m 11s
- Hardware: NVIDIA GeForce RTX 4070 Laptop GPU x1 (CUDA available)
Metrics (Training)
| Metric | Value |
|---|---|
| Training loss (latest) | 4.2869 |
| Training loss (mean) | 3.5010 |
| Training step | 500000 / 500000 |
Limitations & Considerations
- The model is trained with a masked-token diffusion objective and may not behave like an autoregressive LM.
- Data sources may have licensing or content constraints—review source dataset cards before deployment.
- Performance can vary substantially by mode (universal) and prompt structure.