| Run dir : output/_smoke_test_1gpu |
| Log file: output/_smoke_test_1gpu/train.log |
| GPU: NVIDIA H100 NVL | VRAM: 93.1 GiB | PyTorch: 2.11.0+cu130 |
|
|
| Final Configuration: |
| Paths: |
| transformer_path weights/flux2_dev_fp8mixed.safetensors |
| vae_path weights/flux2-vae.safetensors |
| controlnet_path weights/FLUX.2-dev-Fun-Controlnet-Union-2602.safetensors |
| dataset_dir dataset |
| color_map_path configs/color_map.json |
| output_dir output/_smoke_test_1gpu |
| text_encoder_path weights/mistral_3_small_flux2_fp8.safetensors |
| precomputed_embeddings output/text_embeddings_global.pt |
| Model: |
| image_size 1024 |
| num_classes 6 |
| control_in_dim 3072 |
| fusion_dim 768 |
| num_fusion_blocks 3 |
| num_heads 12 |
| num_fourier_bands 32 |
| boundary_threshold 0.1 |
| Training: |
| num_epochs 1 |
| batch_size 4 |
| learning_rate 0.0003 |
| weight_decay 0.01 |
| max_grad_norm 1.0 |
| grad_accum_steps 4 |
| guidance_scale 3.5 |
| num_workers 0 |
| Text Encoder: |
| text_seq_len 512 |
| text_dim 15360 |
| Logging: |
| log_interval 1 |
| save_every_n_epochs 5 |
| val_every_n_epochs 1 |
| WandB: |
| wandb_entity |
| wandb_project _smoke_test_1gpu |
| Resume: |
| resume_from (not set) |
| [MEM @ pre-flight] RAM: 20.7/188.5 GiB (11.0%) | VRAM: 0.0/93.1 GiB (0.0%) |
| |
| ============================================================ |
| [1/8] Text Embeddings |
| ============================================================ |
| Composed prompt (575 chars): |
| Aerial top-down satellite view of American urban area, Google Earth style, 8k resolution, photorealistic satellite imagery, natural daylight, buildings: detaile... |
| |
| === Precomputing Global Text Embedding === |
| Prompt: Aerial top-down satellite view of American urban area, Google Earth style, 8k resolution, photorealistic satellite image... |
| Loading safetensors: weights/mistral_3_small_flux2_fp8.safetensors |
| Building tokenizer from embedded tekken_model ... |
| Detected: 30 layers, hidden=5120, heads=32, kv_heads=8, ffn=32768, vocab=131072 |
| Initialising MistralModel (30 layers)... |
| Dequantizing FP8 weights → bf16 ... |
| Text encoder: 17.3B params, VRAM: 34.69 GB |
| Embedding shape: torch.Size([512, 15360]) |
| Saved global embedding → output/text_embeddings_global.pt |
| Text encoder unloaded. VRAM: 34.7 GB |
| === Global Text Embedding Done === |
| |
| Loaded global text embedding from output/text_embeddings_global.pt (shape: torch.Size([512, 15360])) |
| |
| ============================================================ |
| [2/8] Loading VAE |
| ============================================================ |
| Done (1.1s), VRAM: 0.19 GiB |
| [MEM @ after VAE] RAM: 17.6/188.5 GiB (9.3%) | VRAM: 0.2/93.1 GiB (0.2%) |
| |
| ============================================================ |
| [3/8] Loading Transformer |
| ============================================================ |
| Dequantizing FP8 transformer weights... |
| Dequantized 128 FP8 tensors |
| Converting ComfyUI → diffusers keys... |
| Converted: 331 diffusers keys |
| Loading ControlNet weights... |
| ControlNet: 76 keys |
| Creating Flux2ControlTransformer2DModel (control_in_dim=3072)... |
| Skipped 2 control_img_in keys (dim mismatch): |
| control_img_in.bias [6144] |
| control_img_in.weight [6144, 260] |
| Missing: 2, Unexpected: 0 |
| Initialized control_img_in.weight [6144, 3072] on cuda |
| Initialized control_img_in.bias [6144] on cuda |
| FP8 compression: 203 frozen Linears, 67.9 → 37.9 GiB (saved 30.0 GiB) |
| Done (59.7s), VRAM: 37.90 GiB |
| Gradient checkpointing: enabled |
| Backbone FROZEN: all transformer params set requires_grad=False |
| Gradients will still propagate to HDC²A via control_context autograd |
| [MEM @ after Transformer] RAM: 17.8/188.5 GiB (9.5%) | VRAM: 37.9/93.1 GiB (40.7%) |
| |
| ============================================================ |
| [4/8] Creating HDC²A Adapter |
| ============================================================ |
| HDC²A: 52.4M params |
| Control: 0.0M params |
| Total trainable: 52.4M params |
| |
| ============================================================ |
| [4.5/8] Applying LoRA to ControlNet Control Blocks |
| ============================================================ |
| LoRA rank=32, alpha=32.0, dropout=0 |
| LoRA control_transformer_blocks.0.attn.to_q [6144→6144] |
| LoRA control_transformer_blocks.0.attn.to_k [6144→6144] |
| LoRA control_transformer_blocks.0.attn.to_v [6144→6144] |
| LoRA control_transformer_blocks.0.attn.add_q_proj [6144→6144] |
| LoRA control_transformer_blocks.0.attn.add_k_proj [6144→6144] |
| LoRA control_transformer_blocks.0.attn.add_v_proj [6144→6144] |
| LoRA control_transformer_blocks.0.attn.to_out.0 [6144→6144] |
| LoRA control_transformer_blocks.1.attn.to_q [6144→6144] |
| LoRA control_transformer_blocks.1.attn.to_k [6144→6144] |
| LoRA control_transformer_blocks.1.attn.to_v [6144→6144] |
| LoRA control_transformer_blocks.1.attn.add_q_proj [6144→6144] |
| LoRA control_transformer_blocks.1.attn.add_k_proj [6144→6144] |
| LoRA control_transformer_blocks.1.attn.add_v_proj [6144→6144] |
| LoRA control_transformer_blocks.1.attn.to_out.0 [6144→6144] |
| LoRA control_transformer_blocks.2.attn.to_q [6144→6144] |
| LoRA control_transformer_blocks.2.attn.to_k [6144→6144] |
| LoRA control_transformer_blocks.2.attn.to_v [6144→6144] |
| LoRA control_transformer_blocks.2.attn.add_q_proj [6144→6144] |
| LoRA control_transformer_blocks.2.attn.add_k_proj [6144→6144] |
| LoRA control_transformer_blocks.2.attn.add_v_proj [6144→6144] |
| LoRA control_transformer_blocks.2.attn.to_out.0 [6144→6144] |
| LoRA control_transformer_blocks.3.attn.to_q [6144→6144] |
| LoRA control_transformer_blocks.3.attn.to_k [6144→6144] |
| LoRA control_transformer_blocks.3.attn.to_v [6144→6144] |
| LoRA control_transformer_blocks.3.attn.to_out.0 [6144→6144] |
|
|
| LoRA modules injected: 25 |
| LoRA trainable params: 9.83M |
|
|
| Parameter Statistics: |
| HDC²A Adapter: total=52.4M trainable=52.4M |
| ControlNet (frozen): total=4143.4M LoRA trainable=9.83M |
| Flux2 backbone: total=0.0M trainable=0.0M ✓ |
| ────────────────────────────────────────────────── |
| Total trainable: HDC²A 52.4M + LoRA 9.83M = 62.19M |
|
|
| ============================================================ |
| [5/8] Building Optimizer |
| ============================================================ |
| AdamW: adapter_lr=3.00e-04, backbone_lr=0.00e+00 |
| param_group 'adapter': 112 tensors, lr=3.00e-04 |
| Scheduler: 400 warmup steps → cosine over ~25 steps |
| [6/8] Resume: skipped (no checkpoint specified) |
|
|
| ============================================================ |
| [7/8] Forward Sanity Check |
| ============================================================ |
| [test 1/4] Forward pass (eval mode)... |
| Output shape: torch.Size([1, 4096, 128]) |
| Output stats: mean=0.0481, std=0.5195 |
| VRAM peak (forward): 68.47 GiB |
| [test 2/4] Loss computation (train mode)... |
| Loss value: 1.376319 |
| [test 3/4] Backward pass... |
| Backward completed. VRAM peak (backward): 49.21 GiB |
| [test 4/4] Gradient flow check... |
| HDC²A: 112/112 params have non-zero grad |
| Control: 25/50 params have non-zero grad |
| Top grad norms (HDC²A): |
| semantic_encoder.conv_stem.6.weight: 0.002655 |
| depth_encoder.conv_stem.6.weight: 0.002594 |
| W_d.weight: 0.002533 |
| W_s.weight: 0.002380 |
| fusion_blocks.0.ffn_dep.2.weight: 0.002014 |
| Test result: PASSED |
| [MEM @ after test] RAM: 17.9/188.5 GiB (9.5%) | VRAM: 38.1/93.1 GiB (40.9%) |
| |
| *** --test passed: all models loaded, forward test OK. Exiting. *** |
| |
