---
title: Text-Driven Image Augmentation
emoji: 🖼️
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 5.23.1
python_version: '3.11'
app_file: app.py
startup_duration_timeout: 1h
pinned: false
---

# Text-Driven Image Augmentation

Upload an image and type an instruction. Transforms are selected by **semantic similarity** between your prompt and a catalog of augmentation keywords (1–5 matches above a cosine-similarity threshold).

## How planning works

1. Your instruction is embedded with `sentence-transformers/all-MiniLM-L6-v2`.
2. Each augmentation keyword has several natural-language phrase variants (also pre-embedded).
3. Keywords scoring above the similarity threshold are selected (max 5), with mutual-exclusion rules for opposing effects (e.g. brighten vs darken).

## Supported instructions

**Artistic style transfer** (one style per request; ONNX unless noted)

| Prompt examples | Style | Backend |
|-----------------|-------|---------|
| `cartoon` / `anime style` | Candy (comic/cartoon) | ONNX |
| `mosaic` / `stained glass look` | Mosaic tiles | ONNX |
| `impressionist` / `painterly` | Rain Princess | ONNX |
| `abstract art` / `cubist style` | Udnie | ONNX |
| `pointillism` / `dot art` | Pointillism | ONNX |
| `starry night` / `van gogh starry night` | Starry Night | PyTorch ([ebylmz Space](https://huggingface.co/spaces/ebylmz/fast-neural-style-transfer)) |
| `pencil sketch` / `sketch style` | Sketch | PyTorch (neural if weights available, else classical pencil sketch) |
| `stylize` / `artistic style` | Random style | Picks one of the above |

**Color / classical** (examples — paraphrases work too)
- `rotate randomly`, `rotate left` / `rotate right`, `flip upside down`, `rotate 180 degrees`, `gamma brighten`, `perspective`, `make it brighter`, `more contrast`, `vintage warm look`, `flip horizontally`, `blur softly`, `posterize`, `auto contrast`

**Background replacement** (rembg u2netp + Openverse web search)
- `replace background` — random scenic query
- `replace background with <anything>` — free-form search, e.g. `sunset over Paris`, `neon city at night`
- `replace by <anything>` — same as above
- `replace background with anything` — random query

Backgrounds are CC-licensed photos fetched from Openverse at runtime (no local image files).

**Procedural spatial** (only when prompt mentions cover, cutout, hole, etc.)
- `just cover` — hide a random region (not text-aware)
- `cover and add cutout` — cover plus 1–4 varied cutouts
- `cover and several cutouts avoid text` — covers and cutouts avoid text
- `add cutout avoid text` — 1–4 cutouts placed away from text

## Unsupported

Object insertion (not background swap):

- `put a table behind it`

## Local setup (Windows)

Requires Python 3.10+.

```powershell
cd "C:\Users\Dag Bjornberg\AI\cursor_pilot"
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
python app.py
```

First run downloads the embedding model, RapidOCR, and rembg u2netp weights.

## Models

| Component | Role | License |
|-----------|------|---------|
| `sentence-transformers/all-MiniLM-L6-v2` | Instruction → keyword matching | Apache 2.0 |
| RapidOCR (ONNX) | Text region detection | Apache 2.0 |
| rembg u2netp | Foreground segmentation | MIT |
| ONNX fast neural style (onnxmodelzoo) | Artistic style transfer (5 styles) | See ONNX Model Zoo |
| PyTorch TransformNet (ebylmz Space) | Starry Night style | MIT (Space repo) |
| StyleNet / classical fallback | Pencil sketch style | MIT / classical |

See [backgrounds/README.md](backgrounds/README.md) for background prompt examples.