|
|
--- |
|
|
language: en |
|
|
tags: |
|
|
- mask-predict |
|
|
- diffusion |
|
|
- masked-lm |
|
|
library_name: transformers |
|
|
base_model: answerdotai/ModernBERT-base |
|
|
pipeline_tag: fill-mask |
|
|
--- |
|
|
|
|
|
# modernbert-diffusion-universal |
|
|
|
|
|
## Model Summary |
|
|
A diffusion-style masked language model fine-tuned in `universal` mode using a discrete denoising objective. |
|
|
|
|
|
## Model Details |
|
|
- **Model ID:** philipp-zettl/modernbert-diffusion-universal |
|
|
- **Base model:** answerdotai/ModernBERT-base |
|
|
- **Training mode:** universal |
|
|
- **Task type:** Masked token denoising / diffusion-style infilling |
|
|
|
|
|
## Intended Use |
|
|
Intended as a general-purpose infilling model across text, code, JSON, and chat formats. |
|
|
|
|
|
**Example** |
|
|
```python |
|
|
from refinebert.diffusion_engine import MaskedDiffusionEngine |
|
|
|
|
|
engine = MaskedDiffusionEngine("philipp-zettl/modernbert-diffusion-universal") |
|
|
prompt = "def generate_json(data):" |
|
|
output = engine.generate(prompt, num_new_tokens=25, steps=12, guidance_scale=3.0) |
|
|
print(output) |
|
|
``` |
|
|
|
|
|
## Training Data |
|
|
Datasets are streamed from Hugging Face and mixed by mode. |
|
|
|
|
|
### Dataset Mix |
|
|
| Dataset | Percentage | Purpose | |
|
|
| --- | --- | --- | |
|
|
| HuggingFaceFW/fineweb-edu (sample-10BT) | 40% | General web/edu text | |
|
|
| bigcode/the-stack-dedup (python) | 30% | Python code | |
|
|
| bigcode/the-stack-dedup (json) | 15% | Structured JSON | |
|
|
| HuggingFaceH4/ultrachat_200k (train_sft) | 15% | Instruction chat | |
|
|
|
|
|
Fallbacks: FineWeb-Edu may fall back to Wikitext-103, and The Stack may fall back to CodeParrot depending on availability. |
|
|
|
|
|
## Training Procedure |
|
|
- **Steps:** 500000 |
|
|
- **Batch size:** 16 |
|
|
- **Sequence length:** 256 |
|
|
- **Learning rate:** 5e-05 |
|
|
- **CFG dropout probability:** 0.1 |
|
|
- **Samples loaded into RAM:** 100000 |
|
|
|
|
|
## Training Time & Hardware |
|
|
- **Duration:** 46h 38m 11s |
|
|
- **Hardware:** NVIDIA GeForce RTX 4070 Laptop GPU x1 (CUDA available) |
|
|
|
|
|
## Metrics (Training) |
|
|
| Metric | Value | |
|
|
| --- | --- | |
|
|
| Training loss (latest) | 4.2869 | |
|
|
| Training loss (mean) | 3.5010 | |
|
|
| Training step | 500000 / 500000 | |
|
|
|
|
|
## Limitations & Considerations |
|
|
- The model is trained with a masked-token diffusion objective and may not behave like an autoregressive LM. |
|
|
- Data sources may have licensing or content constraints—review source dataset cards before deployment. |
|
|
- Performance can vary substantially by mode (universal) and prompt structure. |
|
|
|
