README.md

# Indic Heritage Studio v2

**A multimodal content creation tool that reimagines modern photos and prompts through the lens of Indian heritage art forms — running on SDXL + Stable Video Diffusion + ControlNet + per-style LoRAs across 8 × NVIDIA 80GB GPUs (dev) and AMD Radeon Cloud (final demo).**

> Submission for the **AMD AI DevMaster Hackathon (July 15 – August 6, 2026)**
> Track 1: Multimodal AI — Development of Multimodal Content Creation Tools
> Team: **TeamIndicForge**

---

## 🎯 What It Does (v2 — full rebuild)

Indic Heritage Studio v2 transforms everyday inputs into culturally-rooted visual art. Given a text prompt, a reference photo, or both, it produces:

1. **Text → Heritage-styled image** — SDXL 1.0 (DreamShaper-XL turbo) at 1024² with per-style LoRA overlay
2. **Image → Stylized image** — IP-Adapter XL on SDXL with optional multi-style blending (60% Madhubani + 40% Warli)
3. **Image → 4-second video** — Stable Video Diffusion XT 1.1 (25 frames @ 8 fps, 1024×576)
4. **ControlNet composition** — Canny / Depth / OpenPose conditioning for precise composition control
5. **Inpainting** — mask a region, restyle it in any heritage style (heritage restoration use case)
6. **Batch processing** — multi-GPU data parallelism across 4 worker GPUs (~4× speedup)

Each output is styled after one of five authentic Indian art traditions, each with its own **fine-tuned LoRA** trained on real heritage art samples:
- 🟢 **Madhubani** — Bihar folk art, geometric patterns, natural pigments
- 🟤 **Warli** — Maharashtra tribal art, white-on-earth, stick figures
- 🔵 **Pattachitra** — Odisha cloth painting, mythological narratives
- 🟡 **Mughal Miniature** — court painting, fine detail, gold leaf
- 🔴 **Tanjore** — Tamil Nadu, gold-gilded devotional icons

An optional **AI Style Advisor** (powered by AMD's free Qwen API) helps users pick a style and refine prompts — but the **core generation runs 100% on AMD Radeon GPU** in compliance with Track 1 rules.

---

## 🆕 What's New in v2

| Axis | v1 | v2 |
|---|---|---|
| T2I model | SD 1.5 (DreamShaper) | **SDXL 1.0 + DreamShaper-XL turbo** |
| Resolution | 512×512 | **1024×1024 (1536² for showcase)** |
| Style conditioning | Prompt tags only | **Per-style LoRA fine-tune + IP-Adapter XL** |
| Image → video | AnimateDiff 16f | **Stable Video Diffusion XT 1.1, 25 frames** |
| Composition control | ❌ | **ControlNet (Canny/Depth/OpenPose)** |
| Image repair | ❌ | **SDXL inpainting** |
| Batch | Single GPU | **Multi-GPU data parallel, 4 workers** |
| Pipelines | Load/unload on demand | **All 4 resident on dedicated GPUs** |

---

## 🏗️ System Architecture (Multi-GPU)

```
┌─────────────────────────────────────────────────────────────────┐
│                       USER INTERFACE (Gradio)                    │
│  6 tabs + Style Advisor widget + GPU monitor sidebar            │
└──────────────────────────────────┬──────────────────────────────┘
                                   ▼
┌─────────────────────────────────────────────────────────────────┐
│              AGENT LAYER (free AMD Qwen/DeepSeek API)            │
│  StyleAdvisor · PromptEngineer · Critic                         │
└──────────────────────────────────┬──────────────────────────────┘
                                   ▼
┌─────────────────────────────────────────────────────────────────┐
│        CORE GPU LAYER — MULTI-GPU (8 × 80 GB dev)                │
│  GPU 0: SDXL T2I + Inpainting    GPU 4: Batch worker 0          │
│  GPU 1: SDXL + IP-Adapter XL     GPU 5: Batch worker 1          │
│  GPU 2: SVD-XT 1.1 (I2V)         GPU 6: Batch worker 2          │
│  GPU 3: SDXL + ControlNet        GPU 7: Batch worker 3          │
│  Per-style LoRAs (~150 MB each) loaded on demand                │
└─────────────────────────────────────────────────────────────────┘
```

On AMD Radeon Cloud (single GPU), all pipelines share GPU 0 and load/unload on demand. The code auto-detects device count.

---

## 🛠️ Tech Stack

| Layer | Tech |
|---|---|
| Dev hardware | 8 × NVIDIA A100/H100 80GB (640 GB total VRAM) |
| Final demo hardware | AMD Radeon Cloud (Radeon GPU, ~16 GB VRAM) |
| Software stack | ROCm 6.2 (AMD) / CUDA 12.1 (NVIDIA dev) |
| Deep learning | PyTorch 2.4.1 + Diffusers 0.30 + Accelerate 0.34 + PEFT 0.12 |
| T2I model | SDXL 1.0 + DreamShaper-XL turbo |
| Style transfer | IP-Adapter XL (`h94/IP-Adapter`) |
| Image-to-video | Stable Video Diffusion XT 1.1 (`stabilityai/stable-video-diffusion-img2vid-xt-1-1`) |
| Composition control | ControlNet (Canny/Depth/OpenPose SDXL variants) |
| Per-style LoRAs | Trained with PEFT, rank 32, ~800 steps each |
| Agent LLM | Free AMD Model APIs (Qwen3.6-35B-A3B / DeepSeek-V4-Flash) |
| UI | Gradio 4.x |
| Video post | FFmpeg + OpenCV + imageio |
| Image post | Pillow, imageio |

---

## 📦 Installation

### Step 1 — Clone
```bash
git clone https://huggingface.co/Dev2506/indic-heritage-studio
cd indic-heritage-studio
```

# Setup
python -m venv .venv
source .venv/bin/activate
pip install torch==2.4.1 torchvision==0.19.1 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

# Configure
cp .env.example .env
# Edit .env to add AMD_MODEL_API_KEY (free at https://developer.amd.com.cn/radeon/modelapis)

# Launch
python app.py
# Open http://localhost:7860
\`\`\`

---

## 🚀 Usage

### CLI mode

```bash
# Text-to-image in Madhubani style (SDXL + LoRA)
python -m core.text_to_image \
    --prompt "a young woman reading under a banyan tree" \
    --style madhubani \
    --out outputs/t2i_madhubani.png \
    --steps 25 --size 1024 --seed 42

# Style transfer (IP-Adapter XL)
python -m core.style_transfer \
    --image inputs/photo.jpg \
    --style warli \
    --out outputs/styled_warli.png \
    --strength 0.7 --ip-scale 0.7

# Image-to-video (SVD)
python -m core.image_to_video \
    --image outputs/styled_warli.png \
    --out outputs/warli_animated.mp4 \
    --frames 25 --fps 8

# ControlNet (Canny → Mughal)
python -m core.controlnet \
    --condition-image inputs/sketch.png \
    --condition-type canny \
    --prompt "a courtly gathering" \
    --style mughal \
    --out outputs/controlnet_mughal.png

# Inpainting
python -m core.inpainting \
    --image inputs/damaged_painting.jpg \
    --mask inputs/damage_mask.png \
    --prompt "restore the floral border" \
    --style tanjore \
    --out outputs/restored.png

# Batch (multi-GPU, 4 workers)
python -m core.batch_processor \
    --mode style_transfer \
    --input-dir inputs/ \
    --output-dir outputs/batch/ \
    --workers 4
```

### Web UI

```bash
python app.py
```

The UI exposes all 6 modes plus the Style Advisor widget and GPU monitor.

---

## ⚡ Multi-GPU Optimization

The codebase applies these optimizations:

1. **Pipeline-to-GPU pinning** — T2I on GPU 0, IP-Adapter on GPU 1, SVD on GPU 2, ControlNet on GPU 3. No thrash.
2. **Batch data parallelism** — Inputs sharded across GPUs 4-7 via `multiprocessing.spawn`.
3. **Per-style LoRA hot-swap** — `unload_lora_weights` → `load_lora_weights` in ~3 seconds (vs 30+ sec full pipeline reload).
4. **FP16 inference** — `torch_dtype=torch.float16` everywhere.
5. **SDPA attention** — built into PyTorch 2.0+, replaces xformers (CUDA-only).
6. **Attention + VAE slicing** — reduces VRAM peak ~30%.
7. **`torch.inference_mode()`** — all hot paths.
8. **Pipeline singleton reuse** — load once, serve many requests.

### Multi-GPU Benchmark

```bash
python scripts/benchmark.py --configs 1 2 4 8 --samples 4
```

Reports: latency per image, throughput (img/min), peak VRAM, scaling chart.

---

## 📊 Evaluation Rubric Coverage

| Rubric item | Points | How v2 hits it |
|---|---|---|
| Complete input→processing→output workflow | 40 | 6 pipelines: T2I, I2I-style, I2V, ControlNet, Inpaint, batch |
| Innovative creation scenarios | 20 | Indian heritage art × LoRA fine-tuning × ControlNet × SVD = unique vertical |
| Practical application & social value | 20 | Cultural preservation + heritage restoration + accessible creator tool |
| Clear, stable, diverse output on Radeon GPU | 20 | 6 output modes × 5 art styles × LoRA = 30+ high-quality demo outputs |

---

## 🗂️ Project Structure

```
indic-heritage-studio-v2/
├── README.md                       # This file
├── app.py                          # Gradio UI entry point
├── requirements.txt                # Python dependencies
├── .env.example                    # Environment template
├── .gitignore
├── config/
│   ├── settings.py                 # v2 global config (SDXL, multi-GPU, LoRA paths)
│   └── styles.py                   # 5 heritage styles + LoRA scales + motion tuning
├── agents/                         # LLM agent layer (free AMD API)
│   ├── base.py
│   ├── style_advisor.py
│   ├── prompt_engineer.py          # SDXL-aware prompt enrichment
│   └── critic.py                   # Heuristic + LLM critique
├── core/                           # GPU layer
│   ├── text_to_image.py            # SDXL + LoRA + optional refiner
│   ├── style_transfer.py           # IP-Adapter XL + multi-style blending
│   ├── image_to_video.py           # Stable Video Diffusion XT 1.1
│   ├── controlnet.py               # Canny / Depth / OpenPose
│   ├── inpainting.py               # SDXL inpainting + LoRA
│   └── batch_processor.py          # Multi-GPU data parallel
├── training/                       # NEW — LoRA training
│   ├── prepare_dataset.py          # Heritage art dataset prep
│   └── train_lora.py               # PEFT LoRA on SDXL UNet
├── ui/
│   └── gradio_app.py               # 6-tab web UI
├── utils/
│   ├── gpu_utils.py                # Multi-GPU device management
│   ├── image_utils.py
│   └── video_utils.py
├── scripts/
│   ├── benchmark.py                # Multi-GPU scaling benchmark
│   ├── download_models.py          # All v2 models (~35 GB)
│   ├── generate_demo_outputs.py    # Pre-bake demo gallery
│   ├── verify_rocm.py              # Env health check
│   ├── day1_setup.sh               # Day 1 environment setup
│   └── amd_demo_recording.sh       # AMD demo recording script
├── assets/
│   ├── styles/                     # IP-Adapter reference images (5)
│   ├── datasets/                   # LoRA training datasets
│   └── loras/                      # Trained LoRA weights (5 × ~150 MB)
├── docs/
│   ├── architecture.md             # Detailed v2 architecture
│   ├── 2_week_plan.md              # Replan for 8×80GB dev
│   ├── local_dev_setup.md          # NVIDIA CUDA setup
│   └── architecture_diagram.png
├── examples/                       # Demo gallery (pre-baked)
└── tests/                          # Smoke tests
```

---

## 📅 Build Schedule (2-week plan)

See [`docs/2_week_plan.md`](docs/2_week_plan.md) for the day-by-day plan. Summary:

- **Week 1 (Jul 16–22):** Env setup, source heritage art datasets, train 5 LoRAs, smoke-test all 6 pipelines.
- **Week 2 (Jul 23–29):** Pre-bake demo gallery, run multi-GPU benchmark, write PDF + PPT, code freeze.
- **Week 3 (Jul 30–Aug 5):** AMD verification + demo recording + submit. Burns ~1.5 of 10 AMD credits.

---

## 📝 Submission Package

The 4 required deliverables (Track 1):

1. **Project Profile Document (PDF)** — generated via the `pdf` skill in Week 2
2. **Source Code (this repo)** — fork of `AMD-DEV-CONTEST/Radeon-hackathon-2026-07`
3. **Demo Video (3–5 min)** — `rocm-smi` → CLI → live GPU inference → UI walkthrough
4. **PPT / Poster** — generated via the `pptx` skill in Week 2

PR title format: `Track 1, TeamIndicForge, Indic Heritage Studio`

---

## 🆘 Troubleshooting

| Issue | Fix |
|---|---|
| `torch.cuda.is_available() == False` | Reinstall PyTorch with `--index-url https://download.pytorch.org/whl/cu121` |
| LoRA training OOMs | Lower `--rank` from 32 → 16, or `--batch-size` from 1 → 1 with `--grad-accum 8` |
| SVD OOM on AMD | Use `--frames 14` (instead of 25) |
| ControlNet preprocessor download fails | `python -c "from controlnet_aux import CannyDetector; CannyDetector.from_pretrained('lllyasviel/Annotators')"` |
| Multi-GPU batch fails to spawn | Set `MULTI_GPU=0` in `.env` |
| HuggingFace download slow | `HF_HUB_ENABLE_HF_TRANSFER=1` (already in `.env`) |
| Qwen API 429 errors | Agent layer is optional; core tool still works |

---

## 📧 Contact

- **Hackathon email:** ai_dev_contests@amd.com
- **Discord:** https://discord.gg/zt9caur5B3
- **Team:** TeamIndicForg

---

## 📜 License

MIT for project code. Model checkpoints retain their original licenses (see HuggingFace model cards).