Nicholas0228 commited on
Commit
85b56ea
·
verified ·
1 Parent(s): f4acffb

Clean layout: one folder per checkpoint; minimal model card

Browse files
hit-xl16-e600-ema0.9999.safetensors → ImageNet256/HiT-XL16_ep600/model.safetensors RENAMED
File without changes
README.md CHANGED
@@ -1,51 +1,21 @@
1
  ---
 
2
  tags:
3
- - image-generation
4
  - text-to-image
5
- - diffusion
6
- - pixel-space
7
- - imagenet
8
  ---
9
 
10
- # HiT: Dynamic Hierarchical Diffusion Transformer
11
 
12
- HiT generates images directly in pixel space and learns its own visual hierarchy: it groups related
13
- tokens twice and runs most Transformer blocks on the compact groups.
 
14
 
15
- Project page: https://weihangguo.github.io/HiT-webpage/ · Code: https://github.com/Nicholas0228/hnet-jit/tree/claude/hit-clean
 
16
 
17
- ## Checkpoints
18
-
19
- | File | Model | Should reproduce |
20
  |---|---|---|
21
- | `hit-xl16-e600-ema0.9999.safetensors` | HiT-XL/16, ImageNet 256×256, 600 epochs, EMA 0.9999 (959.5M) | FID-50K **1.48** |
22
- | `hit-b16-t2i-sft40k-ema0.99995.safetensors` | HiT-B/16, text-to-image 512×512, 250k pretraining + 40k fine-tuning, EMA 0.99995 (254M) | GenEval 87.78, DPG 85.33, PRISM 58.62, CoRe 45.22, Spatial 52.68 (average **65.93**) |
23
-
24
- Text-to-image training inputs: [Nicholas0228/minit2i-frozen-inputs-v1](https://huggingface.co/datasets/Nicholas0228/minit2i-frozen-inputs-v1),
25
- the same data as MiniT2I, frozen to reproduce our training sequence.
26
-
27
- Each checkpoint comes with its model config and sampling config (`hnet_xl16.json` and
28
- `sampling_hit_xl16_e600.json`; `hit_b16_t2i.json` and `sampling_hit_b16_t2i.json`). The same files are in
29
- the code repository under `configs/`.
30
-
31
- ## ImageNet
32
-
33
- ```bash
34
- torchrun --standalone --nproc-per-node=8 scripts/evaluate_imagenet.py \
35
- --checkpoint hit-xl16-e600-ema0.9999.safetensors --model configs/models/hnet_xl16.json \
36
- --sampling configs/sampling/hit_xl16_e600.json --count 50000 \
37
- --output runs/hit-xl16-e600-fid50k \
38
- --adm-assets "$ADM_ASSETS" --adm-python "$ADM_PYTHON"
39
- ```
40
-
41
- ## Text-to-image
42
-
43
- Needs [google/flan-t5-large](https://huggingface.co/google/flan-t5-large) at revision
44
- `0613663d0d48ea86ba8cb3d7a44f0f65dc596a2a`. Generate each benchmark (`geneval`, `dpg`, `prism`, `corebench`,
45
- `spatial_geneval`), then score it as in `docs/BENCHMARKS.md` of the code repository:
46
 
47
- ```bash
48
- torchrun --standalone --nproc-per-node=8 scripts/benchmarks/generate.py --benchmark geneval \
49
- --checkpoint hit-b16-t2i-sft40k-ema0.99995.safetensors --encoder-root "$FLAN_T5_LARGE" \
50
- --assets "$ASSETS" --output-root runs/t2i-benchmarks
51
- ```
 
1
  ---
2
+ pipeline_tag: unconditional-image-generation
3
  tags:
 
4
  - text-to-image
 
 
 
5
  ---
6
 
7
+ # HiT: Hierarchical Diffusion Transformer for End-to-End Image Generation
8
 
9
+ This repository hosts the official PyTorch checkpoints of HiT. HiT is a pixel-space diffusion Transformer
10
+ that learns to group related image patches into a visual hierarchy and runs most of its blocks on the
11
+ compact groups.
12
 
13
+ - Website: https://nicholas0228.github.io/blog/hit/
14
+ - Code: https://github.com/Nicholas0228/hnet-jit/tree/claude/hit-clean
15
 
16
+ | Checkpoint | Model | Should reproduce |
 
 
17
  |---|---|---|
18
+ | `ImageNet256/HiT-XL16_ep600/model.safetensors` | HiT-XL/16, ImageNet 256×256, 600 epochs | FID-50K 1.48 |
19
+ | `T2I512/HiT-B16_sft40k/model.safetensors` | HiT-B/16, text-to-image 512×512, 40k fine-tuning updates | GenEval 87.78, DPG 85.33, PRISM 58.62, CoRe 45.22, Spatial 52.68 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20
 
21
+ Text-to-image training inputs: [Nicholas0228/minit2i-frozen-inputs-v1](https://huggingface.co/datasets/Nicholas0228/minit2i-frozen-inputs-v1).
 
 
 
 
hit-b16-t2i-sft40k-ema0.99995.safetensors → T2I512/HiT-B16_sft40k/model.safetensors RENAMED
File without changes
hit_b16_t2i.json DELETED
@@ -1,35 +0,0 @@
1
- {
2
- "variant": "hnet",
3
- "image_size": 512,
4
- "patch_size": 16,
5
- "width": 768,
6
- "heads": 12,
7
- "mlp_ratio": 2.6667,
8
- "text_width": 1024,
9
- "text_length": 256,
10
- "text_depth": 2,
11
- "block_depths": [
12
- 3,
13
- 3,
14
- 15,
15
- 3,
16
- 3
17
- ],
18
- "patch_bottleneck": 0,
19
- "thresholds": [
20
- 0.27459803223609924,
21
- 0.24442464113235474
22
- ],
23
- "ratios": [
24
- 2.0,
25
- 2.0
26
- ],
27
- "reader_temperature": 0.05,
28
- "budget_temperature": 0.1,
29
- "budget_weight": 0.01,
30
- "router_init_seed": 1701,
31
- "second_router_init_seed": 1702,
32
- "time_token_init_seed": 5701,
33
- "reference_depth": 17,
34
- "reference_bottleneck": 128
35
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
hnet_xl16.json DELETED
@@ -1,34 +0,0 @@
1
- {
2
- "variant": "hnet",
3
- "image_size": 256,
4
- "patch_size": 16,
5
- "width": 1280,
6
- "heads": 16,
7
- "classes": 1000,
8
- "in_context_len": 8,
9
- "patch_bottleneck": 0,
10
- "reader_temperature": 0.05,
11
- "budget_temperature": 0.1,
12
- "budget_weight": 0.01,
13
- "router_init_seed": 1701,
14
- "second_router_init_seed": 1702,
15
- "context_init_seed": 2701,
16
- "time_token_init_seed": 5801,
17
- "block_depths": [
18
- 10,
19
- 10,
20
- 8,
21
- 10,
22
- 10
23
- ],
24
- "in_context_start": 0,
25
- "proj_dropout": 0.2,
26
- "thresholds": [
27
- 0.8956202864646912,
28
- 0.9993406534194946
29
- ],
30
- "ratios": [
31
- 2,
32
- 2.0
33
- ]
34
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
sampling_hit_b16_t2i.json DELETED
@@ -1,19 +0,0 @@
1
- {
2
- "guidance": 6.5,
3
- "steps": 100,
4
- "a_min": 0.05,
5
- "noise_std": 2.0,
6
- "seed": 0,
7
- "group_guidance": {
8
- "cond": [0.30807870626449585, 0.2524682283401489],
9
- "uncond": [0.24352747201919556, 0.24442464113235474]
10
- },
11
- "reference_scores": {
12
- "geneval": 87.78,
13
- "dpg": 85.33,
14
- "prism": 58.62,
15
- "corebench": 45.22,
16
- "spatial_geneval": 52.68,
17
- "average": 65.93
18
- }
19
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
sampling_hit_xl16_e600.json DELETED
@@ -1,92 +0,0 @@
1
- {
2
- "description": "Final HiT-XL/16 (10/10/8/10/10) epoch-600 sampler: group guidance with a shifted time grid. Thresholds are calibrated for the epoch-600 EMA 0.9999 weights: the conditional prediction uses 1.2x the calibrated group count at every t, the unconditional one 0.5x for t >= 0.85 (stage-1 router only).",
3
- "steps": 50,
4
- "guidance": 2.6,
5
- "interval": [
6
- 0.1,
7
- 1.0
8
- ],
9
- "time_shift": 1.7,
10
- "group_guidance": {
11
- "cond": [
12
- [
13
- 0.0,
14
- 0.1,
15
- 0.9058777987957001,
16
- null
17
- ],
18
- [
19
- 0.1,
20
- 0.2,
21
- 0.9078725427389145,
22
- null
23
- ],
24
- [
25
- 0.2,
26
- 0.3,
27
- 0.9099719127019247,
28
- null
29
- ],
30
- [
31
- 0.3,
32
- 0.4,
33
- 0.9130289283665743,
34
- null
35
- ],
36
- [
37
- 0.4,
38
- 0.5,
39
- 0.9159530401229858,
40
- null
41
- ],
42
- [
43
- 0.5,
44
- 0.6,
45
- 0.9204822778701782,
46
- null
47
- ],
48
- [
49
- 0.6,
50
- 0.7,
51
- 0.9296736717224121,
52
- null
53
- ],
54
- [
55
- 0.7,
56
- 0.8,
57
- 0.9439890384674072,
58
- null
59
- ],
60
- [
61
- 0.8,
62
- 0.9,
63
- 0.9644635915756226,
64
- null
65
- ],
66
- [
67
- 0.9,
68
- 1.01,
69
- 0.9895514249801637,
70
- null
71
- ]
72
- ],
73
- "uncond": [
74
- [
75
- 0.85,
76
- 0.9,
77
- 0.815306822458903,
78
- null
79
- ],
80
- [
81
- 0.9,
82
- 1.01,
83
- 0.7975938022136688,
84
- null
85
- ]
86
- ]
87
- },
88
- "reference_fid50k": {
89
- "official": 1.4778,
90
- "imagenet_val": 2.7332
91
- }
92
- }