Lucy-2

A 112.5M-parameter text-to-image diffusion transformer, pretrained from scratch at 128x128 on a single RTX 3060.

Not a fine-tune of Stable Diffusion or anything else — the transformer weights start from random init. Only the VAE and the CLIP text encoder are pretrained off-the-shelf components, both frozen throughout.

Usage

You must pass the negative prompt. It is not optional.

python inference.py --prompt "a portrait of a woman smiling" --n 4

# the important part
NEGATIVE = "watermark, text overlay"

Recommended settings: cfg_scale 4.5, steps 50.

CC3M is largely stock photography, so the model learned to draw caption bars. Training tagged those captions with watermark, text overlay so the sampler can steer away from the concept. Without the negative prompt you get watermarks and noticeably worse faces — watermarked stock imagery correlates with the waxy portraits in the dataset, so steering away from it lifts quality generally.

Architecture

Parameters	112.5M (transformer only)
Resolution	128x128
Latent	16x16x4, stabilityai/sd-vae-ft-mse (f=8), frozen
Text encoder	openai/clip-vit-large-patch14, frozen
Backbone	DiT, dim 640, depth 12, 10 heads, patch 1 (256 tokens)
Conditioning	adaLN-Zero on timestep + pooled text; cross-attention to text tokens
Objective	Rectified flow, logit-normal timestep sampling
Sampler	Euler ODE with classifier-free guidance

Training

Roughly 44 GPU-hours total on one RTX 3060 (12 GB), in stages:

stage	data	time
Pretrain	CC3M, 3.02M pairs, 2.86 epochs, 8.6M samples	20 h
Fine-tune	watermark-filtered subset	4 h
Fine-tune	watermark-tagged pool (4% marked) + 10% CelebA face crops	12 h

The face crops matter: in a scene a face occupies maybe 5x5 of a 16x16 latent, but a portrait crop that fills the frame gets the whole budget. Mixing them in fixed faces within 2500 steps after they had failed for the entire pretrain.

Limitations

This is a small model on a small compute budget — about 2 H100-hours equivalent, against the ~150,000 A100-hours behind Stable Diffusion. Calibrate accordingly.

Works: scenes, composition, colour, material, lighting. Landscapes, interiors, food, vehicles, single animals. Close-up portraits.

Fails:

• Multi-subject prompts. "Two cats sitting on a sofa" produces uncanny hybrid creatures. Counting is unreliable. This never improved at any stage.
• Relational clauses. "Strawberries next to a glass of milk" drops the milk.
• Structured objects. "A blue bicycle" gives a blue object, not a bicycle — thin spoked geometry is beyond a 16x16 latent.
• Text rendering. No.
• Proper nouns. CC3M hypernymises named entities ("person", "actor"), so they were never learnable.
• Faces in wide scenes remain poor; only close-ups improved.

Training data and licensing

Trained on CC3M (via dimzhead/cc3m-256) and CelebA (via tglcourse/CelebA-faces-cropped-128).

CelebA is licensed for non-commercial research use only. Since those images were used in training, this model is released for research use only. Do not use it commercially. CC3M images are likewise not owned by the dataset authors and carry their own terms.

The training data is web-scraped and unfiltered beyond the watermark heuristic described above. It carries the biases of that corpus, including in what kinds of people and scenes it represents well. The CelebA mixture skews the portrait distribution toward that dataset's demographics. No safety filtering of any kind was applied to outputs.