| --- |
| title: Text-Driven Image Augmentation |
| emoji: 🖼️ |
| colorFrom: blue |
| colorTo: green |
| sdk: gradio |
| sdk_version: 5.23.1 |
| python_version: '3.11' |
| app_file: app.py |
| startup_duration_timeout: 1h |
| pinned: false |
| --- |
| |
| # Text-Driven Image Augmentation |
|
|
| Upload an image and type an instruction. Transforms are selected by **semantic similarity** between your prompt and a catalog of augmentation keywords (1–5 matches above a cosine-similarity threshold). |
|
|
| ## How planning works |
|
|
| 1. Your instruction is embedded with `sentence-transformers/all-MiniLM-L6-v2`. |
| 2. Each augmentation keyword has several natural-language phrase variants (also pre-embedded). |
| 3. Keywords scoring above the similarity threshold are selected (max 5), with mutual-exclusion rules for opposing effects (e.g. brighten vs darken). |
|
|
| ## Supported instructions |
|
|
| **Artistic style transfer** (one style per request; ONNX unless noted) |
|
|
| | Prompt examples | Style | Backend | |
| |-----------------|-------|---------| |
| | `cartoon` / `anime style` | Candy (comic/cartoon) | ONNX | |
| | `mosaic` / `stained glass look` | Mosaic tiles | ONNX | |
| | `impressionist` / `painterly` | Rain Princess | ONNX | |
| | `abstract art` / `cubist style` | Udnie | ONNX | |
| | `pointillism` / `dot art` | Pointillism | ONNX | |
| | `starry night` / `van gogh starry night` | Starry Night | PyTorch ([ebylmz Space](https://huggingface.co/spaces/ebylmz/fast-neural-style-transfer)) | |
| | `pencil sketch` / `sketch style` | Sketch | PyTorch (neural if weights available, else classical pencil sketch) | |
| | `stylize` / `artistic style` | Random style | Picks one of the above | |
|
|
| **Color / classical** (examples — paraphrases work too) |
| - `rotate randomly`, `rotate left` / `rotate right`, `flip upside down`, `rotate 180 degrees`, `gamma brighten`, `perspective`, `make it brighter`, `more contrast`, `vintage warm look`, `flip horizontally`, `blur softly`, `posterize`, `auto contrast` |
|
|
| **Background replacement** (rembg u2netp + Openverse web search) |
| - `replace background` — random scenic query |
| - `replace background with <anything>` — free-form search, e.g. `sunset over Paris`, `neon city at night` |
| - `replace by <anything>` — same as above |
| - `replace background with anything` — random query |
|
|
| Backgrounds are CC-licensed photos fetched from Openverse at runtime (no local image files). |
|
|
| **Procedural spatial** (only when prompt mentions cover, cutout, hole, etc.) |
| - `just cover` — hide a random region (not text-aware) |
| - `cover and add cutout` — cover plus 1–4 varied cutouts |
| - `cover and several cutouts avoid text` — covers and cutouts avoid text |
| - `add cutout avoid text` — 1–4 cutouts placed away from text |
|
|
| ## Unsupported |
|
|
| Object insertion (not background swap): |
|
|
| - `put a table behind it` |
|
|
| ## Local setup (Windows) |
|
|
| Requires Python 3.10+. |
|
|
| ```powershell |
| cd "C:\Users\Dag Bjornberg\AI\cursor_pilot" |
| python -m venv .venv |
| .\.venv\Scripts\Activate.ps1 |
| pip install -r requirements.txt |
| python app.py |
| ``` |
|
|
| First run downloads the embedding model, RapidOCR, and rembg u2netp weights. |
|
|
| ## Models |
|
|
| | Component | Role | License | |
| |-----------|------|---------| |
| | `sentence-transformers/all-MiniLM-L6-v2` | Instruction → keyword matching | Apache 2.0 | |
| | RapidOCR (ONNX) | Text region detection | Apache 2.0 | |
| | rembg u2netp | Foreground segmentation | MIT | |
| | ONNX fast neural style (onnxmodelzoo) | Artistic style transfer (5 styles) | See ONNX Model Zoo | |
| | PyTorch TransformNet (ebylmz Space) | Starry Night style | MIT (Space repo) | |
| | StyleNet / classical fallback | Pencil sketch style | MIT / classical | |
|
|
| See [backgrounds/README.md](backgrounds/README.md) for background prompt examples. |
|
|
