README.md

---
title: Minecraftify
emoji: ⚡
colorFrom: blue
colorTo: red
sdk: gradio
sdk_version: 6.18.0
python_version: '3.13'
app_file: app.py
pinned: false
license: apache-2.0
short_description: Mincraftify converts all images into mc-style LIVE!
tags:
  - track:wood
  - sponsor:openai
  - sponsor:modal
  - achievement:offgrid
  - achievement:welltuned
  - achievement:offbrand
  - achievement:fieldnotes
---
# Minecraftify!

**Minecraftify your images live.**

Minecraftify is a Hugging Face Gradio Space that turns uploaded photos into a faithful vanilla Minecraft interpretation of the same scene. It is powered by a fine-tuned **FLUX.2-Klein-4B** img2img LoRA trained on a custom dataset generated with **Qwen-Edit-25-12**.

## Live Demo

* **Space:** Minecraftify
* **Demo video:** [YouTube walkthrough](https://youtu.be/W-2yEjlTOK4)
* **Blog post:** [Project blog](https://huggingface.co/blog/build-small-hackathon/minecraftify)

## Current Status

Playable demo Space with:

* still-image Minecraftification
* live webcam mode
* persistent model caching on Space storage
* LoRA-based FLUX.2-Klein inference
* Gradio UI with image upload, webcam input, and advanced settings

## What It Does

Minecraftify transforms an input image into a Minecraft-style version of the same scene while trying to preserve:

* composition
* camera angle
* layout
* objects already present in the scene
* color relationships and overall structure

The model is tuned to:

* replace realistic surfaces with Minecraft blocks and voxel geometry
* keep the scene recognizable
* avoid unnecessary scene changes
* convert people, animals, and objects into Minecraft-style equivalents where needed

## Project Artifacts

* **Base model:** `black-forest-labs/FLUX.2-klein-4B`
* **LoRA adapter:** `AnimeOverlord/flux2-klein-4b-mc-v2`
* **Dataset:** 376 image pairs created with Qwen-Edit-25-12
* **Training script:** `train_dreambooth_lora_flux2_klein_img2img.py`

## Fine-Tuning Setup

The LoRA was trained with FLUX.2-Klein img2img using a paired dataset with:

* `source_image` as the conditioning image
* `edited_image` as the target image
* `prompt_used` as the caption column

Training highlights:

* `train_batch_size=1`
* `gradient_accumulation_steps=4`
* `mixed_precision=bf16`
* `learning_rate=2e-6`
* `lr_scheduler=constant_with_warmup`
* `lr_warmup_steps=200`
* `max_train_steps=1200`
* `rank=64`
* `cache_latents`
* `use_8bit_adam`
* `aspect_ratio_buckets` enabled

## Hackathon Fit

Minecraftify is designed as a compact, fun, image-to-image Space with a strong visual identity and an immediate demo loop.

It fits the small-model spirit because the core generation path is built around a **4B FLUX Klein model** with a LoRA adapter rather than a large general-purpose model.

## How to Demo

1. Upload an image or start the webcam.
2. Choose still image or live mode.
3. Adjust inference steps, guidance scale, and seed.
4. Click **Minecraftify!**
5. Download or inspect the result.

## Recommended Demo Settings

* **Inference steps:** 3
* **Guidance scale:** 3.0
* **Seed:** any fixed value for reproducibility
* **Input:** well-lit images with clear objects and simple scenes

## Features

* image upload
* webcam capture
* live frame processing
* prompt-based scene preservation
* persistent model caching in Hugging Face Space storage
* adjustable inference settings
* output comparison view

## Model and Runtime

The app loads the FLUX.2-Klein base model and then applies the Minecraft LoRA adapter.

Runtime behavior:

* models are cached on persistent Space storage
* weights are reused across runs when present
* the pipeline is kept in memory for the active session
* image generation uses img2img inference for scene preservation

## Space Storage

This Space is configured to use persistent storage so model files do not need to be downloaded every time the Space restarts.

* model cache path: `/data/models`
* Hugging Face cache path: `/data/.huggingface`

## Architecture

**Input image** → **FLUX.2-Klein img2img** → **Minecraft LoRA** → **Rendered output**

For live mode, webcam frames are captured continuously and only the latest frame is processed when the model becomes available.

## Local Development

This project was trained locally with PyTorch and Accelerate, and the training workflow also supports pushing the fine-tuned model to the Hugging Face Hub.

### Running locally with PyTorch

#### 1) Install the training dependencies

For the most up-to-date Diffusers example scripts, it is recommended to install Diffusers from source:

```bash
git clone https://github.com/huggingface/diffusers
cd diffusers
pip install -e .
````

Then install the FLUX DreamBooth example requirements:

```bash
cd examples/dreambooth
pip install -r requirements_flux.txt
```

#### 2) Configure Accelerate

Set up Accelerate for your environment:

```bash
accelerate config
```

If you want the default configuration without answering prompts:

```bash
accelerate config default
```

If you are running in a notebook or another environment without an interactive shell:

```python
from accelerate.utils import write_basic_config
write_basic_config()
```

If possible, enable `torch compile` in Accelerate for faster training. Also make sure `peft>=0.6.0` is installed, since PEFT is used as the LoRA backend.

### Training FLUX.2-Klein LoRA on an image-to-image dataset

This project uses the FLUX 2 Klein 4B base model and trains a LoRA adapter on a paired img2img dataset.

```bash
cd diffusers/examples/dreambooth && accelerate launch train_dreambooth_lora_flux2_klein_img2img.py \
  --pretrained_model_name_or_path=black-forest-labs/FLUX.2-klein-4B \
  --output_dir="flux2-i2i" \
  --dataset_name="AnimeOverlord/mine-dataset" \
  --image_column="edited_image" \
  --cond_image_column="source_image" \
  --caption_column="prompt_used" \
  --gradient_checkpointing \
  --cache_latents \
  --train_batch_size=1 \
  --guidance_scale=1 \
  --gradient_accumulation_steps=4 \
  --mixed_precision="bf16" \
  --optimizer="prodigy" \
  --learning_rate=1 \
  --lr_warmup_steps=200 \
  --max_train_steps=1200 \
  --rank=64 \
  --seed="0" \
  --push_to_hub \
  --hub_model_id="[YOURACCOUNT]/flux2-klein-4b-mc" \
  --aspect_ratio_buckets="672,1568;688,1504;720,1456;752,1392;800,1328;832,1248;880,1184;944,1104;1024,1024;1104,944;1184,880;1248,832;1328,800;1392,752;1456,720;1504,688;1568,672"
```

### Notes

* The dataset contains **376 images** created with **Qwen-Edit-25-12**.
* The training run uses a paired img2img setup with:

  * `source_image` as the conditioning image
  * `edited_image` as the target image
  * `prompt_used` as the caption
* `push_to_hub` uploads the trained LoRA adapter to the Hugging Face Hub.
* The aspect-ratio buckets help keep training efficient across different image shapes.

### Output

After training, the LoRA adapter is published to:

```bash
YOURACCOUNT/flux2-klein-4b-mc
```

## Blog and Video Links

* **Blog:** [Read the build notes](https://huggingface.co/blog/build-small-hackathon/minecraftify)
* **YouTube:** [Watch the walkthrough](https://youtu.be/W-2yEjlTOK4)
* **LinkedIn Post** [Read the social media post](https://www.linkedin.com/posts/md-abdul-kalam-khan_ai-machinelearning-generativeai-ugcPost-7472386161223680000-PdaN/)

## Link to Notebooks Used
* **Training Notbook:** [Modal Notebook](https://modal.com/notebooks/kalamkhan-se/main/nb-ygLQVGDvJR3FbrNpQwGYfV)
* **Data Creation Notebook:** [Modal Notebook](https://modal.com/notebooks/kalamkhan-se/main/nb-L22FAz1tYTB39h7tXkXF4N)

## Credits

* Base model: Black Forest Labs
* Fine-tuning workflow: Hugging Face Diffusers
* Dataset creation: Qwen-Edit-25-12
* UI: Gradio

## License

This project is a demo Space for experimentation and presentation. Check the model and dataset licenses before redistribution.