|
|
--- |
|
|
license: apache-2.0 |
|
|
pipeline_tag: text-generation |
|
|
datasets: |
|
|
- codeparrot/github-code-clean |
|
|
- bigcode/starcoderdata |
|
|
- bigcode/the-stack-smol |
|
|
tags: |
|
|
- diffusion |
|
|
- llm |
|
|
- diffreaper |
|
|
- dllm |
|
|
- mercury |
|
|
language: |
|
|
- en |
|
|
--- |
|
|
# DiffReaper 3 |
|
|
|
|
|
DiffReaper 3 is the third revision of DiffReaper; a experimental 1.5B parameter Discrete Diffusion Language Model (dLLM) designed for high-throughput parallel token prediction. |
|
|
Unlike traditional autoregressive models, DiffReaper is optimized for non-linear sequence refinement across mixed Python logic and natural language corpora. |
|
|
|
|
|
## Model Details |
|
|
- **Architecture:** 24-Layer Transformer Encoder |
|
|
- **Hidden Dimension:** 2048 |
|
|
- **Attention Heads:** 16 |
|
|
- **Objective:** Discrete Masked Diffusion (Mercury-style) |
|
|
- **Training Precision:** BF16 |
|
|
- **Context Window:** 1024 tokens |
