Step 5000: Initial Foundational Pre-training Weight Drop

README.md

@@ -0,0 +1,19 @@
+# DiffReaper-Talk
+
+A 1.5B parameter Discrete Diffusion Language Model (dLLM) optimized for parallel token prediction. Trained during foundational pre-training phase on general text corpora.
+
+## Summary
+DiffReaper-Talk uses a Transformer-based discrete diffusion architecture to predict multiple tokens in parallel. This approach avoids the sequential bottleneck of standard autoregressive generation.
+
+## Technical Details
+- **Architecture:** 24-Layer Transformer Encoder
+- **Embedding Dim:** 2048
+- **Heads:** 16
+- **Parameters:** ~1.5 Billion
+- **Hardware:** 1x NVIDIA A100 (80GB VRAM)
+- **Objective:** Markovian Discrete Denoising (Continuous Embedding Space)
+- **Precision:** Mixed BF16
+- **Context Window:** 1024 Tokens
+
+## Current Status
+Foundational pre-training active. Logic and domain-specific training (Code) to be applied post-convergence.