Hugging Face's logo

montignyp
/
MYai

Model card Files
xet
 Community

YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Flux Identity LoRA Training Environment

A complete training environment for creating identity LoRAs using FLUX.1 Dev on RunPod with NVIDIA A100 80GB.

Overview

This project provides everything needed to:

  • Train identity-preserving LoRAs on FLUX.1 Dev
  • Validate and prepare training datasets
  • Test LoRAs in ComfyUI
  • Scale to ControlNet and IP-Adapter (future)

Target Configuration

  • GPU: NVIDIA A100 80GB
  • Base Model: FLUX.1 Dev
  • Training Precision: bf16
  • Image Resolution: 1024x1024
  • Dataset Size: ~1,000 identity images
  • Training Toolkit: kohya_ss/sd-scripts

Folder Structure 

/workspace/flux-project/
├── models/                    # Model files
│   ├── flux/                  # FLUX.1 Dev base model
│   ├── lora/                  # Trained LoRA outputs
│   ├── vae/                   # VAE (ae.safetensors)
│   ├── clip/                  # Text encoders (CLIP-L, T5-XXL)
│   ├── controlnet/            # ControlNet models (future)
│   └── ip-adapter/            # IP-Adapter models (future)
├── datasets/
│   └── identity/
│       ├── images/            # Raw training images
│       └── processed/         # Processed images for training
├── output/                    # Training outputs & checkpoints
├── scripts/                   # Utility scripts
├── logs/                      # TensorBoard logs
└── configs/                   # Training configurations

Quick Start

1. Initial Setup

Run the setup script on RunPod:

cd /workspace/flux-project/scripts
chmod +x setup_environment.sh
./setup_environment.sh

This will:

  • Install system dependencies
  • Create Python virtual environment
  • Install PyTorch with CUDA support
  • Install ComfyUI and sd-scripts
  • Set up model symlinks

2. Download Models

After setup, download the required models:

# Login to Hugging Face
huggingface-cli login

# Download FLUX.1 Dev (requires accepting license)
huggingface-cli download black-forest-labs/FLUX.1-dev \
    --local-dir /workspace/flux-project/models/flux

# Download CLIP-L
huggingface-cli download openai/clip-vit-large-patch14 \
    --local-dir /workspace/flux-project/models/clip/clip-l

# Download T5-XXL
huggingface-cli download google/t5-v1_1-xxl \
    --local-dir /workspace/flux-project/models/clip/t5-xxl

3. Prepare Dataset

Place your images in /workspace/flux-project/datasets/identity/images/

4. Validate Dataset 

python /workspace/flux-project/scripts/validate_dataset.py

5. Start Training 

/workspace/flux-project/scripts/start_training.sh

Dataset Preparation Workflow

Image Requirements

Requirement Recommended Minimum
Resolution 1024x1024 512x512
Format PNG, JPG Any supported
Quality High, uncompressed No artifacts
Count 500-2000 100
Variety Multiple angles, lighting, expressions At least 3 poses

Image Guidelines

  1. Consistency: Same subject across all images
  2. Quality: Sharp, well-lit, no motion blur
  3. Variety: Different:
    • Angles (front, 3/4, profile)
    • Lighting conditions
    • Expressions
    • Backgrounds (ideally varied)
    • Clothing (for person identity)
  4. Cropping: Subject should fill 50-80% of frame
  5. Resolution: Higher is better (will be resized)

Caption Format

Each image needs a matching .txt file with the same name:

photo_001.jpg  →  photo_001.txt
photo_002.png  →  photo_002.txt

Caption Best Practices

  1. Use a trigger word: Start every caption with a unique identifier

    ohwx person, a man with short brown hair, wearing a blue shirt

  2. Be descriptive: Include details about pose, clothing, setting

    ohwx person, a woman smiling, professional headshot, studio lighting, grey background

  3. Consistent structure: Keep caption format similar across dataset

    [trigger], [subject description], [pose/action], [setting], [lighting]

  4. Avoid:

    • Overly long captions (aim for 20-50 words)
    • Inconsistent terminology
    • Subjective qualities ("beautiful", "amazing")

Trigger Word Strategy

Choose a trigger word that is:

  • Unique: Not a common word (ohwx, sks, zwx)
  • Pronounceable: Easy to remember
  • Consistent: Used identically in every caption

Example trigger words: ohwx, sks, xyz123, [subject]

Running Validation 

# Full validation with duplicate detection
python /workspace/flux-project/scripts/validate_dataset.py /workspace/flux-project/datasets/identity/images

# Quick validation (skip duplicates)
python /workspace/flux-project/scripts/validate_dataset.py --no-duplicates /path/to/images

# Save report to file
python /workspace/flux-project/scripts/validate_dataset.py -o report.txt /path/to/images

Dataset Statistics 

python /workspace/flux-project/scripts/dataset_stats.py /workspace/flux-project/datasets/identity/images

Training Workflow

Configuration

Edit /workspace/flux-project/configs/train_config.toml:

Key parameters to adjust:

Parameter Default Description
max_train_epochs 10 Number of training epochs
learning_rate 1e-4 Base learning rate
network_dim 32 LoRA rank (8-128)
network_alpha 16 LoRA alpha (typically dim/2)
train_batch_size 1 Images per step
gradient_accumulation_steps 4 Effective batch multiplier

Starting Training 

# Using default config
/workspace/flux-project/scripts/start_training.sh

# Using custom config
/workspace/flux-project/scripts/start_training.sh /path/to/custom_config.toml

Monitoring Progress

TensorBoard logs are saved to /workspace/flux-project/logs/:

tensorboard --logdir /workspace/flux-project/logs --bind_all --port 6006

Access at: http://<your-runpod-ip>:6006

Checkpoints

Checkpoints are saved to /workspace/flux-project/output/:

  • identity_lora-epoch-01.safetensors
  • identity_lora-epoch-02.safetensors
  • ...
  • identity_lora.safetensors (final)

Recommended Training Schedule

Dataset Size Epochs Steps (approx)
100 images 20-30 500-750
500 images 10-15 1250-1875
1000 images 8-12 2000-3000
2000 images 5-8 2500-4000

LoRA Export Workflow

Checkpoint Files

After training, your LoRA files will be in:

/workspace/flux-project/output/
├── identity_lora.safetensors          # Final model
├── identity_lora-epoch-*.safetensors  # Epoch checkpoints
└── training_state/                    # Resume state (if enabled)

Testing Checkpoints

Different epochs may produce different quality results. Test multiple:

  1. Copy checkpoint to LoRA folder:

    cp /workspace/flux-project/output/identity_lora-epoch-05.safetensors \
   /workspace/flux-project/models/lora/

  2. Load in ComfyUI and generate test images

  3. Compare quality across epochs

Converting Checkpoints (if needed)

sd-scripts outputs are already in safetensors format. No conversion needed.

Inference Workflow

Starting ComfyUI 

/workspace/flux-project/scripts/start_comfyui.sh

Access at: http://<your-runpod-ip>:8188

Loading LoRA in ComfyUI

  1. In ComfyUI, add a "Load LoRA" node
  2. Select your trained LoRA from the dropdown
  3. Connect to your Flux workflow
  4. Set LoRA strength (start with 0.7-1.0)

Recommended Inference Settings

Setting Value
LoRA Strength 0.7 - 1.0
CFG Scale 3.5 - 7.0
Steps 20 - 30
Sampler euler, dpmpp_2m
Scheduler normal, karras

Using Your Trigger Word

Include your trigger word in prompts:

ohwx person, professional portrait photo, studio lighting,
sharp focus, 8k resolution

Experiment with:

  • Trigger word position (start vs middle)
  • Combining with style keywords
  • Different prompt lengths

Future Expansion

ControlNet Integration

Download Flux ControlNet when ready:

# InstantX Flux ControlNet Union
huggingface-cli download InstantX/FLUX.1-dev-Controlnet-Union \
    --local-dir /workspace/flux-project/models/controlnet/flux-controlnet-union

Use cases:

  • Pose control
  • Depth guidance
  • Canny edge control

IP-Adapter Setup

Download IP-Adapter for identity preservation:

huggingface-cli download InstantX/FLUX.1-dev-IP-Adapter \
    --local-dir /workspace/flux-project/models/ip-adapter/flux-ip-adapter

Use cases:

  • Reference image guidance
  • Style transfer
  • Character consistency

Troubleshooting

Common Issues

CUDA Out of Memory

  • Reduce train_batch_size to 1
  • Enable gradient_checkpointing
  • Enable cache_latents_to_disk
  • Reduce network_dim

Training Loss Not Decreasing

  • Check caption quality
  • Try lower learning rate (5e-5)
  • Verify images are properly formatted
  • Check for corrupt images with validation script

Blurry/Distorted Results

  • Train longer (more epochs)
  • Increase network_dim
  • Verify dataset quality
  • Check for duplicate images

ComfyUI Can't Find LoRA

  • Check symlinks are correct
  • Verify file is in .safetensors format
  • Restart ComfyUI

Getting Help

File Reference

File Purpose
scripts/setup_environment.sh Complete environment setup
scripts/start_comfyui.sh Launch ComfyUI server
scripts/start_training.sh Launch LoRA training
scripts/validate_dataset.py Validate training images
scripts/dataset_stats.py Generate dataset statistics
configs/train_config.toml Training configuration

License

This project setup is provided as-is for educational purposes.

Model licenses:

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support