|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- darwinkernelpanic/DiffReaper-3 |
|
|
--- |
|
|
# DiffReaper-Talk |
|
|
|
|
|
A 1.5B parameter Discrete Diffusion Language Model (dLLM) optimized for parallel token prediction. Trained during foundational pre-training phase on general text corpora. |
|
|
|
|
|
## Summary |
|
|
DiffReaper-Talk uses a Transformer-based discrete diffusion architecture to predict multiple tokens in parallel. This approach avoids the sequential bottleneck of standard autoregressive generation. |
|
|
|
|
|
## Technical Details |
|
|
- **Architecture:** 24-Layer Transformer Encoder |
|
|
- **Embedding Dim:** 2048 |
|
|
- **Heads:** 16 |
|
|
- **Parameters:** ~1.5 Billion |
|
|
- **Hardware:** 1x NVIDIA A100 (80GB VRAM) |
|
|
- **Objective:** Markovian Discrete Denoising (Continuous Embedding Space) |
|
|
- **Precision:** Mixed BF16 |
|
|
- **Context Window:** 1024 Tokens |
|
|
|
|
|
## Current Status |
|
|
Phase 2 (Logic) Complete. Logic and domain-specific training (Code) to be applied post-convergence. |
