File size: 3,559 Bytes
8d0cde4 1f74f4d 8d0cde4 ed5c7e7 1f74f4d 8d0cde4 ed5c7e7 8d0cde4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 | ---
license: mit
tags:
- diffusion
- ddpm
- text-to-image
- pytorch
- fashion-mnist
- classifier-free-guidance
library_name: pytorch
pipeline_tag: text-to-image
---
# text-diffusion-fashion-mnist
A **text-conditioned diffusion model built from scratch in PyTorch** — a miniature Stable
Diffusion trained on Fashion-MNIST (32×32 grayscale). A U-Net learns to reverse a Gaussian
noising process, conditioned on **frozen CLIP text embeddings**, and generates a garment image
from a caption using **classifier-free guidance**.
📦 **Code / training / full write-up:** https://github.com/mbsdeepak/text-diffusion-fashion-mnist
*One row per class (t-shirt, trouser, pullover, dress, coat, sandal, shirt, sneaker, bag, ankle
boot); each image is generated from pure noise, DDIM 50 steps, guidance 1.5.*
## What it does
Give it one of the 10 Fashion-MNIST categories and it synthesises a brand-new image of that
item from random noise:
```
"sneaker" ──► [model] ──► a novel 32×32 image of a sneaker
```
## Files
| File | Description |
|------|-------------|
| `model.safetensors` | U-Net weights (19.4M params, raw / non-EMA) |
| `config.json` | The `Config` used to build the U-Net |
## How to load & sample
The architecture is defined in the [GitHub repo](https://github.com/mbsdeepak/text-diffusion-fashion-mnist),
so load the weights into it:
```bash
git clone https://github.com/mbsdeepak/text-diffusion-fashion-mnist
cd text-diffusion-fashion-mnist
pip install -r requirements.txt
huggingface-cli download mbsdeepak/text-diffusion-fashion-mnist model.safetensors --local-dir .
```
```python
import torch
from safetensors.torch import load_file
from config import get_config, FASHION_CLASSES
from src.unet import UNet
from src.diffusion import GaussianDiffusion
from src.text_encoder import TextConditioner
from src.data import denormalize
from torchvision.utils import save_image
cfg = get_config()
model = UNet(cfg).to(cfg.device)
model.load_state_dict(load_file("model.safetensors"))
model.eval()
cond = TextConditioner(cfg).to(cfg.device)
diff = GaussianDiffusion(cfg).to(cfg.device)
labels = torch.arange(len(FASHION_CLASSES), device=cfg.device) # one of each class
imgs = diff.ddim_sample(model, cond, labels)
save_image(denormalize(imgs), "out.png", nrow=len(FASHION_CLASSES))
```
## Training details
- **Data:** Fashion-MNIST, 32×32, normalized to [-1, 1]
- **Objective:** ε-prediction MSE (DDPM), cosine noise schedule, T=1000
- **Conditioning:** frozen CLIP (`openai/clip-vit-base-patch32`) text embeddings via FiLM +
cross-attention; 15% caption dropout for classifier-free guidance
- **Trained:** 25 epochs on Apple Silicon (MPS), final loss ≈ 0.042
- **Sampling:** DDIM, 50 steps, guidance scale 1.5 (low guidance — high CFG artifacts on this
small, short-trained model)
> These are the **raw** weights, not EMA — for a short (25-epoch) run the EMA average still lags
> the live weights, so the raw model produces the cleaner samples.
## Limitations
- Conditioned on a **fixed set of 10 captions**, so it's text-driven class-conditional
generation, not open-vocabulary text-to-image (the cross-attention plumbing is the same; the
dataset is the limit).
- 32×32 grayscale — a learning/portfolio project to understand the method, not photorealism.
## References
Ho et al. *DDPM* (2020) · Nichol & Dhariwal *Improved DDPM* (2021) · Song et al. *DDIM* (2021) ·
Ho & Salimans *Classifier-Free Guidance* (2022) · Rombach et al. *Latent Diffusion* (2022).
|