README.md

metadata

title: Minecraftify
emoji: ⚡
colorFrom: blue
colorTo: red
sdk: gradio
sdk_version: 6.18.0
python_version: '3.13'
app_file: app.py
pinned: false
license: apache-2.0
short_description: Mincraftify converts all images into mc-style LIVE!
tags:
  - track:wood
  - sponsor:openai
  - sponsor:modal
  - achievement:offgrid
  - achievement:welltuned
  - achievement:offbrand
  - achievement:fieldnotes

Minecraftify!

Minecraftify your images live.

Minecraftify is a Hugging Face Gradio Space that turns uploaded photos into a faithful vanilla Minecraft interpretation of the same scene. It is powered by a fine-tuned FLUX.2-Klein-4B img2img LoRA trained on a custom dataset generated with Qwen-Edit-25-12.

Live Demo

• Space: Minecraftify
• Demo video: YouTube walkthrough
• Blog post: Project blog

Current Status

Playable demo Space with:

• still-image Minecraftification
• live webcam mode
• persistent model caching on Space storage
• LoRA-based FLUX.2-Klein inference
• Gradio UI with image upload, webcam input, and advanced settings

What It Does

Minecraftify transforms an input image into a Minecraft-style version of the same scene while trying to preserve:

• composition
• camera angle
• layout
• objects already present in the scene
• color relationships and overall structure

The model is tuned to:

• replace realistic surfaces with Minecraft blocks and voxel geometry
• keep the scene recognizable
• avoid unnecessary scene changes
• convert people, animals, and objects into Minecraft-style equivalents where needed

Project Artifacts

• Base model: black-forest-labs/FLUX.2-klein-4B
• LoRA adapter: AnimeOverlord/flux2-klein-4b-mc-v2
• Dataset: 376 image pairs created with Qwen-Edit-25-12
• Training script: train_dreambooth_lora_flux2_klein_img2img.py

Fine-Tuning Setup

The LoRA was trained with FLUX.2-Klein img2img using a paired dataset with:

• source_image as the conditioning image
• edited_image as the target image
• prompt_used as the caption column

Training highlights:

• train_batch_size=1
• gradient_accumulation_steps=4
• mixed_precision=bf16
• learning_rate=2e-6
• lr_scheduler=constant_with_warmup
• lr_warmup_steps=200
• max_train_steps=1200
• rank=64
• cache_latents
• use_8bit_adam
• aspect_ratio_buckets enabled

Hackathon Fit

Minecraftify is designed as a compact, fun, image-to-image Space with a strong visual identity and an immediate demo loop.

It fits the small-model spirit because the core generation path is built around a 4B FLUX Klein model with a LoRA adapter rather than a large general-purpose model.

How to Demo

1. Upload an image or start the webcam.
2. Choose still image or live mode.
3. Adjust inference steps, guidance scale, and seed.
4. Click Minecraftify!
5. Download or inspect the result.

Recommended Demo Settings

• Inference steps: 3
• Guidance scale: 3.0
• Seed: any fixed value for reproducibility
• Input: well-lit images with clear objects and simple scenes

Features

• image upload
• webcam capture
• live frame processing
• prompt-based scene preservation
• persistent model caching in Hugging Face Space storage
• adjustable inference settings
• output comparison view

Model and Runtime

The app loads the FLUX.2-Klein base model and then applies the Minecraft LoRA adapter.

Runtime behavior:

• models are cached on persistent Space storage
• weights are reused across runs when present
• the pipeline is kept in memory for the active session
• image generation uses img2img inference for scene preservation

Space Storage

This Space is configured to use persistent storage so model files do not need to be downloaded every time the Space restarts.

• model cache path: /data/models
• Hugging Face cache path: /data/.huggingface

Architecture

Input image → FLUX.2-Klein img2img → Minecraft LoRA → Rendered output

For live mode, webcam frames are captured continuously and only the latest frame is processed when the model becomes available.

Local Development

This project was trained locally with PyTorch and Accelerate, and the training workflow also supports pushing the fine-tuned model to the Hugging Face Hub.

Running locally with PyTorch

1) Install the training dependencies

For the most up-to-date Diffusers example scripts, it is recommended to install Diffusers from source:

git clone https://github.com/huggingface/diffusers
cd diffusers
pip install -e .

Then install the FLUX DreamBooth example requirements:

cd examples/dreambooth
pip install -r requirements_flux.txt

2) Configure Accelerate

Set up Accelerate for your environment:

accelerate config

If you want the default configuration without answering prompts:

accelerate config default

If you are running in a notebook or another environment without an interactive shell:

from accelerate.utils import write_basic_config
write_basic_config()

If possible, enable torch compile in Accelerate for faster training. Also make sure peft>=0.6.0 is installed, since PEFT is used as the LoRA backend.

Training FLUX.2-Klein LoRA on an image-to-image dataset

This project uses the FLUX 2 Klein 4B base model and trains a LoRA adapter on a paired img2img dataset.

cd diffusers/examples/dreambooth && accelerate launch train_dreambooth_lora_flux2_klein_img2img.py \
  --pretrained_model_name_or_path=black-forest-labs/FLUX.2-klein-4B \
  --output_dir="flux2-i2i" \
  --dataset_name="AnimeOverlord/mine-dataset" \
  --image_column="edited_image" \
  --cond_image_column="source_image" \
  --caption_column="prompt_used" \
  --gradient_checkpointing \
  --cache_latents \
  --train_batch_size=1 \
  --guidance_scale=1 \
  --gradient_accumulation_steps=4 \
  --mixed_precision="bf16" \
  --optimizer="prodigy" \
  --learning_rate=1 \
  --lr_warmup_steps=200 \
  --max_train_steps=1200 \
  --rank=64 \
  --seed="0" \
  --push_to_hub \
  --hub_model_id="[YOURACCOUNT]/flux2-klein-4b-mc" \
  --aspect_ratio_buckets="672,1568;688,1504;720,1456;752,1392;800,1328;832,1248;880,1184;944,1104;1024,1024;1104,944;1184,880;1248,832;1328,800;1392,752;1456,720;1504,688;1568,672"

Notes

• The dataset contains 376 images created with Qwen-Edit-25-12.

• The training run uses a paired img2img setup with:

  • source_image as the conditioning image
  • edited_image as the target image
  • prompt_used as the caption

• push_to_hub uploads the trained LoRA adapter to the Hugging Face Hub.

• The aspect-ratio buckets help keep training efficient across different image shapes.

Output

After training, the LoRA adapter is published to:

YOURACCOUNT/flux2-klein-4b-mc

Blog and Video Links

• Blog: Read the build notes
• YouTube: Watch the walkthrough
• LinkedIn Post Read the social media post

Link to Notebooks Used

• Training Notbook: Modal Notebook
• Data Creation Notebook: Modal Notebook

Credits

• Base model: Black Forest Labs
• Fine-tuning workflow: Hugging Face Diffusers
• Dataset creation: Qwen-Edit-25-12
• UI: Gradio

License

This project is a demo Space for experimentation and presentation. Check the model and dataset licenses before redistribution.