metadata
license: apache-2.0
language:
- en
base_model:
- darwinkernelpanic/DiffReaper-3
DiffReaper-Talk
A 1.5B parameter Discrete Diffusion Language Model (dLLM) optimized for parallel token prediction. Trained during foundational pre-training phase on general text corpora.
Summary
DiffReaper-Talk uses a Transformer-based discrete diffusion architecture to predict multiple tokens in parallel. This approach avoids the sequential bottleneck of standard autoregressive generation.
Technical Details
- Architecture: 24-Layer Transformer Encoder
- Embedding Dim: 2048
- Heads: 16
- Parameters: ~1.5 Billion
- Hardware: 1x NVIDIA A100 (80GB VRAM)
- Objective: Markovian Discrete Denoising (Continuous Embedding Space)
- Precision: Mixed BF16
- Context Window: 1024 Tokens
Current Status
Phase 2 (Logic) Complete. Logic and domain-specific training (Code) to be applied post-convergence.