README.md

metadata

title: Text-Driven Image Augmentation
emoji: 🖼️
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 5.23.1
python_version: '3.11'
app_file: app.py
startup_duration_timeout: 1h
pinned: false

Text-Driven Image Augmentation

Upload an image and type an instruction. Transforms are selected by semantic similarity between your prompt and a catalog of augmentation keywords (1–5 matches above a cosine-similarity threshold).

How planning works

1. Your instruction is embedded with sentence-transformers/all-MiniLM-L6-v2.
2. Each augmentation keyword has several natural-language phrase variants (also pre-embedded).
3. Keywords scoring above the similarity threshold are selected (max 5), with mutual-exclusion rules for opposing effects (e.g. brighten vs darken).

Supported instructions

Artistic style transfer (one style per request; ONNX unless noted)

Prompt examples	Style	Backend
cartoon / anime style	Candy (comic/cartoon)	ONNX
mosaic / stained glass look	Mosaic tiles	ONNX
impressionist / painterly	Rain Princess	ONNX
abstract art / cubist style	Udnie	ONNX
pointillism / dot art	Pointillism	ONNX
starry night / van gogh starry night	Starry Night	PyTorch (ebylmz Space)
pencil sketch / sketch style	Sketch	PyTorch (neural if weights available, else classical pencil sketch)
stylize / artistic style	Random style	Picks one of the above

Color / classical (examples — paraphrases work too)

• rotate randomly, rotate left / rotate right, flip upside down, rotate 180 degrees, gamma brighten, perspective, make it brighter, more contrast, vintage warm look, flip horizontally, blur softly, posterize, auto contrast

Background replacement (rembg u2netp + Openverse web search)

• replace background — random scenic query
• replace background with <anything> — free-form search, e.g. sunset over Paris, neon city at night
• replace by <anything> — same as above
• replace background with anything — random query

Backgrounds are CC-licensed photos fetched from Openverse at runtime (no local image files).

Procedural spatial (only when prompt mentions cover, cutout, hole, etc.)

• just cover — hide a random region (not text-aware)
• cover and add cutout — cover plus 1–4 varied cutouts
• cover and several cutouts avoid text — covers and cutouts avoid text
• add cutout avoid text — 1–4 cutouts placed away from text

Unsupported

Object insertion (not background swap):

• put a table behind it

Local setup (Windows)

Requires Python 3.10+.

cd "C:\Users\Dag Bjornberg\AI\cursor_pilot"
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
python app.py

First run downloads the embedding model, RapidOCR, and rembg u2netp weights.

Models

Component	Role	License
sentence-transformers/all-MiniLM-L6-v2	Instruction → keyword matching	Apache 2.0
RapidOCR (ONNX)	Text region detection	Apache 2.0
rembg u2netp	Foreground segmentation	MIT
ONNX fast neural style (onnxmodelzoo)	Artistic style transfer (5 styles)	See ONNX Model Zoo
PyTorch TransformNet (ebylmz Space)	Starry Night style	MIT (Space repo)
StyleNet / classical fallback	Pencil sketch style	MIT / classical

See backgrounds/README.md for background prompt examples.