Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,32 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
+
|
| 5 |
+
# TinyImageGen
|
| 6 |
+
|
| 7 |
+
TinyImageGen is a small, fast unconditional image generation model. It features a total of 639k parameters and was trained on 50k images from DataComp for 15 epochs. While image models aren't our primary focus at FromZero, we decided to create our very first one.
|
| 8 |
+
|
| 9 |
+
## Architecture
|
| 10 |
+
|
| 11 |
+
TinyImageGen uses a custom architecture inspired by our text-to-text models, featuring mHC, Hadamard FFNs with SwiGLU intervals, 2D Axial RoPE, and a continuous diffusion objective.
|
| 12 |
+
|
| 13 |
+
- Hidden Size: `96`
|
| 14 |
+
- Hidden Layers: `6`
|
| 15 |
+
- Patch Size: `2×2`
|
| 16 |
+
- Attention Heads: `4`
|
| 17 |
+
- KV Heads: `2` (Grouped-Query Attention)
|
| 18 |
+
- Intermediate Size (for SwiGLU): `160`
|
| 19 |
+
- SwiGLU Interval: `3` (every 3rd layer)
|
| 20 |
+
- Number of Lanes: `4`
|
| 21 |
+
- RoPE Theta: `2500.0`
|
| 22 |
+
|
| 23 |
+
This architecture allows TinyImageGen to remain fast and parameter-efficient while still providing the effective depth and width of a much larger model.
|
| 24 |
+
|
| 25 |
+
## Training
|
| 26 |
+
|
| 27 |
+
As stated above, we trained TinyImageGen on 50k images from DataComp for 15 epochs.
|
| 28 |
+
|
| 29 |
+
- Final Loss: `0.2119`
|
| 30 |
+
|
| 31 |
+
## Generated Sample
|
| 32 |
+
|