Create README.md

5c1f255 11 days ago

6.17 kB

	---
	license: mit
	datasets:
	- aipracticecafe/anime-faces-256px-v2
	language:
	- en
	pipeline_tag: text-to-image
	---
	# Toy-diffusion Ablation Results & Benchmark Analysis


	Following the results presented in the [i1 paper](https://huggingface.co/datasets/aipracticecafe/anime-faces-256px) we conducted a series of architectural controlled experiments on our [aipracticecafe/anime-faces-256px-v2](https://huggingface.co/datasets/aipracticecafe/anime-faces-256px-v2) dataset (comprising 87,505 high-quality face crops) using Flux 2 VAE latents and using the [Toy-diffusion](https://github.com/deeplearningcafe/toy-diffusion) training code.

	Both Single and Dual stream dits completely remove the ADA-LN conditioning, instead the timestep is projected as a token and prepended to the text tokens. Training uses AdamW 8bit optimizer with warmup-constant learning rate of 4e-4 and a batch size of 256.

	Below is the detailed benchmark performance FID-20K (Fréchet Inception Distance, ) across our evaluated checkpoints.

	### 1. Unified Leaderboard of Evaluated Checkpoints

	\| Checkpoint Name \| Configuration & Parameters \| Epoch \| FID (↓) \| Performance Summary \|
	\| :--- \| :--- \| :---: \| :---: \| :--- \|
	\| `dual_stream_768_12_trans_2_bs_1024` \| Width: 768, Depth: 12, `trans_2` Adapter, BS: 1024 \| 1400 \| 2.4240 \| Best Overall Configuration. Large-batch training combined with long training yields the best performance. \|
	\| `dual_stream_768_12_trans_2` \| Width: 768, Depth: 12, `trans_2` Adapter, BS: 512 \| 600 \| 3.0635 \| Original baseline. \|
	\| `dual_stream_768_12_llm_te_log_normal` \| Width: 768, Depth: 12, LLM Text Encoder, Log-Normal \| 1000 \| 3.1219 \| Good prompt following which makes the model prone to instabilities for poorly represented tags. \|
	\| `dual_stream_768_12_modern_cfm` \| Width: 768, Depth: 12, `trans_2` Adapter, Contrastive FM \| 700 \| 3.2707 \| Modern text adapter and CFM doesn't seem to provide improvements overbaseline as wall time is similar. \|
	\| `efficient_unet_bs_256` \| SnapGen Conv-Attn Hybrid, BS: 256 \| 400 \| 3.4479 \| CNN-Transformer hybrid. Fast early-stage convergence at the cost of more than double wall time training and more vram consumption. \|
	\| `single_stream_768_14_trans_2` \| Width: 768, Depth: 14, Single-Stream Backbone, BS: 512 \| 600 \| 3.8954 \| Slower training throughput. Struggles to drop below FID 3.7. \|
	\| `sprint_dual_stream_768_13_trans_2_fined` \| SPRINT (Sparse), Finetuned Dense on Full Sequence \| 1000 \| 3.9647 \| Recovers remarkably well during dense finetuning, preserving visual fidelity while cutting pretraining costs. \|
	\| `dual_stream_768_16_mlp` \| Width: 768, Depth: 16, simple MLP text adapter \| 400 \| 4.2485 \| Bottlenecked heavily by the weak MLP text adapter. \|
	\| `sprint_single_stream_768_14_fined` \| SPRINT Single-Stream, Finetuned Dense \| 1000 \| 7.3793 \| Does not recover completely from sparse training, lagging significantly behind dual-stream variants. \|
	\| `sprint_dual_stream_768_13_trans_2` \| SPRINT (Sparse), No Finetuning (75% drop ratio) \| 700 \| 13.8281 \| High sparse degradation. Evaluating on dense inputs without dense finetuning is not recommended. \|
	\| `sprint_single_stream_768_14_trans_img_2_rope_text` \| SPRINT (Sparse), No Finetuning (75% drop ratio) \| 750 \| 9.7705 \| Lower fid than dual-stream version. \|

	---

	### Architectural Insights

	#### Text adapter
	Our results provide clear, empirical validation of the importantece of a strong text encoder adapter:
	* Comparing `dual_stream_768_16_mlp` (depth 16, FID 4.57 at 500 epochs) to `dual_stream_768_12_trans_2` (depth 12, FID 3.54 at 500 epochs) reveals a clear pattern. Investing capacity into a 2-layer Transformer Text Adapter (`trans_2`) rather than raw MLP produces a massive visual upgrade.

	#### B. Single-Stream vs. Dual-Stream Backbones
	* Training Efficiency: Dual-stream blocks completed training epochs in ~3.5 minutes, whereas Single-stream blocks required ~5.1 minutes—under identical VRAM constraints (~23GB). By processing text and image tokens independently, Dual-stream models seem to be more `hardware-friendly`.
	* Visual Fidelity: Dual-stream models drop FID significantly faster and settle at a much lower ultimate floor (FID `3.06` vs. Single-stream's `3.69` plateau).
	* The Overfitting & Cosplay Tradeoff: We observed a distinct qualitative divergence:
	* Dual-stream models are prone to localized concept overfitting (e.g., struggling to render Asuka in Misato's uniform, consistently over-associating characters with their native outfits).
	* Single-stream models handle complex conditional blending and "cosplay" prompts with far better zero-shot composition, indicating stronger cross-modality token integration.

	#### C. SPRINT: Efficient Sparse Pre-Training
	* The sparse token-dropping framework (SPRINT) with a 75% drop ratio achieved a ~50% reduction in training time and lowered active GPU memory requirements from 23GB to 17GB. Even with the Finetuning stage, the overall training wall time is shorter than the dense version.
	* Finetuning recovers performance: Evaluating SPRINT models directly results in highly degraded FID scores (FID ~13.8) due to the dense-sparse distribution shift. However, following the dense finetuning recipe (`_fined`), the model recovers rapidly, achieving an FID of `3.96` by epoch 1000.
	* Dual Stream is favored: Despite the good FID scores exhibited by the Single stream architecture, during the finetuning stage only the dual stream architecture is able to recover the high performance.

	Developed by: [aipracticecafe](https://huggingface.co/aipracticecafe)

	## Content Warning & Rights
	- NSFW Content: This dataset contains sensitive and NSFW (Not Safe For Work) material. It is intended for research and generative modeling purposes.
	- Legal Disclaimer: I do not own the rights to any of the images in this dataset. All images are the property of their respective creators and were scraped from Danbooru. This dataset is provided for educational and research purposes under fair use.

	---
	license: mit
	datasets:
	- aipracticecafe/anime-faces-256px-v2
	language:
	- en
	pipeline_tag: text-to-image
	---
	# Toy-diffusion Ablation Results & Benchmark Analysis


	Following the results presented in the [i1 paper](https://huggingface.co/datasets/aipracticecafe/anime-faces-256px) we conducted a series of architectural controlled experiments on our [aipracticecafe/anime-faces-256px-v2](https://huggingface.co/datasets/aipracticecafe/anime-faces-256px-v2) dataset (comprising 87,505 high-quality face crops) using Flux 2 VAE latents and using the [Toy-diffusion](https://github.com/deeplearningcafe/toy-diffusion) training code.

	Both Single and Dual stream dits completely remove the ADA-LN conditioning, instead the timestep is projected as a token and prepended to the text tokens. Training uses AdamW 8bit optimizer with warmup-constant learning rate of 4e-4 and a batch size of 256.

	Below is the detailed benchmark performance FID-20K (Fréchet Inception Distance, ) across our evaluated checkpoints.

	### 1. Unified Leaderboard of Evaluated Checkpoints

	\| Checkpoint Name \| Configuration & Parameters \| Epoch \| FID (↓) \| Performance Summary \|
	\| :--- \| :--- \| :---: \| :---: \| :--- \|
	\| `dual_stream_768_12_trans_2_bs_1024` \| Width: 768, Depth: 12, `trans_2` Adapter, BS: 1024 \| 1400 \| 2.4240 \| Best Overall Configuration. Large-batch training combined with long training yields the best performance. \|
	\| `dual_stream_768_12_trans_2` \| Width: 768, Depth: 12, `trans_2` Adapter, BS: 512 \| 600 \| 3.0635 \| Original baseline. \|
	\| `dual_stream_768_12_llm_te_log_normal` \| Width: 768, Depth: 12, LLM Text Encoder, Log-Normal \| 1000 \| 3.1219 \| Good prompt following which makes the model prone to instabilities for poorly represented tags. \|
	\| `dual_stream_768_12_modern_cfm` \| Width: 768, Depth: 12, `trans_2` Adapter, Contrastive FM \| 700 \| 3.2707 \| Modern text adapter and CFM doesn't seem to provide improvements overbaseline as wall time is similar. \|
	\| `efficient_unet_bs_256` \| SnapGen Conv-Attn Hybrid, BS: 256 \| 400 \| 3.4479 \| CNN-Transformer hybrid. Fast early-stage convergence at the cost of more than double wall time training and more vram consumption. \|
	\| `single_stream_768_14_trans_2` \| Width: 768, Depth: 14, Single-Stream Backbone, BS: 512 \| 600 \| 3.8954 \| Slower training throughput. Struggles to drop below FID 3.7. \|
	\| `sprint_dual_stream_768_13_trans_2_fined` \| SPRINT (Sparse), Finetuned Dense on Full Sequence \| 1000 \| 3.9647 \| Recovers remarkably well during dense finetuning, preserving visual fidelity while cutting pretraining costs. \|
	\| `dual_stream_768_16_mlp` \| Width: 768, Depth: 16, simple MLP text adapter \| 400 \| 4.2485 \| Bottlenecked heavily by the weak MLP text adapter. \|
	\| `sprint_single_stream_768_14_fined` \| SPRINT Single-Stream, Finetuned Dense \| 1000 \| 7.3793 \| Does not recover completely from sparse training, lagging significantly behind dual-stream variants. \|
	\| `sprint_dual_stream_768_13_trans_2` \| SPRINT (Sparse), No Finetuning (75% drop ratio) \| 700 \| 13.8281 \| High sparse degradation. Evaluating on dense inputs without dense finetuning is not recommended. \|
	\| `sprint_single_stream_768_14_trans_img_2_rope_text` \| SPRINT (Sparse), No Finetuning (75% drop ratio) \| 750 \| 9.7705 \| Lower fid than dual-stream version. \|

	---

	### Architectural Insights

	#### Text adapter
	Our results provide clear, empirical validation of the importantece of a strong text encoder adapter:
	* Comparing `dual_stream_768_16_mlp` (depth 16, FID 4.57 at 500 epochs) to `dual_stream_768_12_trans_2` (depth 12, FID 3.54 at 500 epochs) reveals a clear pattern. Investing capacity into a 2-layer Transformer Text Adapter (`trans_2`) rather than raw MLP produces a massive visual upgrade.

	#### B. Single-Stream vs. Dual-Stream Backbones
	* Training Efficiency: Dual-stream blocks completed training epochs in ~3.5 minutes, whereas Single-stream blocks required ~5.1 minutes—under identical VRAM constraints (~23GB). By processing text and image tokens independently, Dual-stream models seem to be more `hardware-friendly`.
	* Visual Fidelity: Dual-stream models drop FID significantly faster and settle at a much lower ultimate floor (FID `3.06` vs. Single-stream's `3.69` plateau).
	* The Overfitting & Cosplay Tradeoff: We observed a distinct qualitative divergence:
	* Dual-stream models are prone to localized concept overfitting (e.g., struggling to render Asuka in Misato's uniform, consistently over-associating characters with their native outfits).
	* Single-stream models handle complex conditional blending and "cosplay" prompts with far better zero-shot composition, indicating stronger cross-modality token integration.

	#### C. SPRINT: Efficient Sparse Pre-Training
	* The sparse token-dropping framework (SPRINT) with a 75% drop ratio achieved a ~50% reduction in training time and lowered active GPU memory requirements from 23GB to 17GB. Even with the Finetuning stage, the overall training wall time is shorter than the dense version.
	* Finetuning recovers performance: Evaluating SPRINT models directly results in highly degraded FID scores (FID ~13.8) due to the dense-sparse distribution shift. However, following the dense finetuning recipe (`_fined`), the model recovers rapidly, achieving an FID of `3.96` by epoch 1000.
	* Dual Stream is favored: Despite the good FID scores exhibited by the Single stream architecture, during the finetuning stage only the dual stream architecture is able to recover the high performance.

	Developed by: [aipracticecafe](https://huggingface.co/aipracticecafe)

	## Content Warning & Rights
	- NSFW Content: This dataset contains sensitive and NSFW (Not Safe For Work) material. It is intended for research and generative modeling purposes.
	- Legal Disclaimer: I do not own the rights to any of the images in this dataset. All images are the property of their respective creators and were scraped from Danbooru. This dataset is provided for educational and research purposes under fair use.