README.md

metadata

title: Rune Goblin
colorFrom: purple
colorTo: gray
sdk: docker
app_port: 7860
pinned: false

🪄 Rune Goblin

Draw bad spells, suffer beautifully.

🙏 Credits

• 🤖 Made with Codex — coding and some asset creation (the BOSS and rune_goblin magic-circle assets are Codex-generated).
• 🧠 Models by OpenBMB — openbmb/MiniCPM-V-4.6 (fine-tuned spell engine + base dialogue/story).
• ⚡ Thanks to Modal for fine-tuning credits.

Full attributions, including game art packs, are in CREDITS.md.

A tiny dungeon crawler where players draw spells in an invented symbolic language (RuneLang) and a fine-tuned openbmb/MiniCPM-V-4.6 acts as the spell engine — reading glyph combinations and emitting JSON that drives attacks, curses and game-state changes. Runtime visuals are not image-generated; spell metadata recolors, resizes, retargets and animates existing/procedural game assets.

See rune_goblin_plan.md for the full design doc.

Architecture

Rune buttons (Gradio / React)
   → serialized rune sequence + game state
   → fine-tuned openbmb/MiniCPM-V-4.6 + LoRA   (rune_goblin.inference)
   → spell outcome JSON              (validated by rune_goblin.schema)
   → game state engine               (rune_goblin.game, clamps HP)
   → updated UI

Drawn canvas (Gradio vision app)
   → canvas image + game state
   → goblinV1-gguf Q4_K_M sketch reader
   → visual_reading / rune metadata  (rune_goblin.vision_inference)
   → fine-tuned RuneLang spell JSON  (validated by rune_goblin.schema)
   → asset / VFX planner             (rune_goblin.vfx — rule-based today, MiniCPM-V-4.6 later)
   → attack / VFX metadata           (palette, projectile, particles, screen shake)
   → CSS/canvas renderer + game state engine

For the playable MVP the asset planner is a small deterministic function (rune_goblin.vfx) instead of a second model: it is instant and never fails, and its metadata shape mirrors the plan so a real MiniCPM-V-4.6 planner can be swapped in later without touching the renderer.

The deterministic rule engine (rune_goblin.engine) is both the dataset oracle (it generates training targets) and the runtime fallback (used until a fine-tuned adapter exists, so the UIs run before training finishes).

The RPG sandbox (app/rpg_app.py + src/rune_goblin/world.py) realizes the map-exploration direction: tile-map areas with enemies, NPCs, chests, shrines, powerups, locked doors and a boss. The canvas client owns movement/rendering; resolve_world_cast turns drawn/selected runes + the faced target into a validated spell and a list of world actions (unlock, loot, defeat, heal, travel-gate…). Python stays the spell engine and balance authority.

Layout

Path	What
src/rune_goblin/runelang.py	16 runes, grammar, combos, enemies, rooms
src/rune_goblin/engine.py	spell-physics rule engine + HP clamping
src/rune_goblin/schema.py	Pydantic spell schema + JSON repair
src/rune_goblin/generate_dataset.py	synthetic dataset → data/*.jsonl
src/rune_goblin/finetune.py	LoRA/QLoRA training (TRL + PEFT)
src/rune_goblin/inference.py	load base + adapter, cast spells
src/rune_goblin/vision_inference.py	load ASHu2/goblinV1, read canvas drawings
src/rune_goblin/evaluate.py	game-engine eval metrics
src/rune_goblin/vfx.py	deterministic asset/VFX planner (palette, projectile, particles, shake)
src/rune_goblin/game.py	5-room dungeon state machine (vision + text/rule cast paths)
src/rune_goblin/world.py	RPG world: tile-map areas, entities, and cast→world-action resolution
app/rpg_app.py	Gradio + canvas RPG — free-roaming sandbox (the main game)
app/rpg_bridge.py	FastAPI /rg/world + /rg/cast bridge for the canvas client
app/rpg_static/	rpg.js + rpg.css — the HTML5 canvas game client
app/rpg_static/sprites/	curated Tiny Swords sprites (CC0) baked for the renderer
assets/	raw Tiny Swords packs (git-ignored; drop the itch download here)
app/vision_app.py	Gradio — linear 5-room combat game (draw + tap, VFX, rune guide)
app/app.py	minimal Gradio rune-button game (legacy)
api/server.py	FastAPI backend for the React frontend
frontend/	Vite + React custom cursed-dungeon UI
notebooks/	dataset exploration and Modal.com vision fine-tuning notebooks
data/	drop your .jsonl training data here
models/	downloaded base model + trained adapters

Setup

Prerequisites: uv, Node 20+, an HF_TOKEN in .env.

# 1. Python env (UI + game only)
uv sync

# 2. Add the heavy ML stack for training / local model inference
uv sync --extra train

# 2b. Add GGUF runtime support for the local vision model
uv sync --extra gguf

# 3. Download the fine-tuning base model into models/
uv run rune-goblin-download            # add --gguf for the .env serving model

# 4. Frontend deps
cd frontend && npm install && cd ..

Workflow

# Generate a synthetic dataset (writes data/rune_spells.jsonl + _val.jsonl)
uv run rune-goblin-data --n 5000

# (You can also just drop your own data/*.jsonl — same chat format.)

# Fine-tune LoRA on your GPU (QLoRA 4-bit by default; 16GB is plenty)
uv run rune-goblin-train --data data/rune_spells.jsonl

# Evaluate (rule engine, or the model once trained)
uv run rune-goblin-eval --n 200            # add --rules-only to skip the model

Vision fine-tuning on Modal.com

For the hand-drawn canvas model, use the Modal.com notebook workflow:

1. Open notebooks/MODAL-run-p1Jun2026.ipynb in Modal Notebooks.
2. Attach a GPU, add your Hugging Face and W&B secrets, and run the notebook top-to-bottom.
3. The notebook prepares the vision JSONL data, fine-tunes openbmb/MiniCPM-V-4.6 with ms-swift LoRA, evaluates samples, and exports the adapter artifacts.

See docs/FINETUNE_VISION.md for the full fine-tuning reference and model-serving notes.

Run the game

🗺️ RPG sandbox (the main game)

A free-roaming campaign RPG. You start on a character-select title screen and pick one of five goblin heroes (Warrior / Rogue / Poison Rogue / Hunter / Barbarian, art from assets/Goblin Pack #1), each with its own HP, courage and rune affinities. Then explore nine connected areas — Toll Road hub, Mirror Fungus Caverns, Wet Library, Bone Market, Clock Sewer, Gate Approach, the Calendar Beast Arena, plus the Frostbite Pass and Ember Foundry (a linked ice/forge loop off the hub) — fighting enemies, gaining XP and levels, buying weapons that modify your spells, helping or scaring NPCs, and leaving durable story flags that steer one of four endings (Broken / Repaired / Devoured / secret Tollmaster). The final boss has three phases and your hero can evolve into the Goblin King.

Two models power the game: the fine-tuned goblinV1 vision model reads your drawn runes, and the base openbmb/MiniCPM-V-4.6-gguf model drives interactive NPC dialogue (press T to talk, or cast at an NPC). Movement and rendering run in an HTML5 canvas; casting and dialogue round-trip to Python, which stays the balance authority. The game is served at /play.

RG_USE_MODEL=1 \
RG_VISION_MODEL=models/goblinV1-gguf/gguf/rune-goblin-v46-Q4_K_M.gguf \
RG_VISION_MMPROJ=models/goblinV1-gguf/gguf/rune-goblin-v46-mmproj-f16.gguf \
RG_USE_DIALOGUE_MODEL=1 \
RG_DIALOGUE_MODEL=models/MiniCPM-V-4.6-gguf/MiniCPM-V-4_6-Q4_K_M.gguf \
uv run --extra gguf python app/rpg_app.py
# → http://localhost:7862   (set RG_USE_MODEL=0 to play purely on the rule engine)

Admin mode. A fully-unlocked world: every area-to-area portal and locked door is walkable, locked chests open with any cast, and the hero holds the key story items (Calendar Key, Debt Receipt, Thawed Ember…). When it's on, a 🔓 ADMIN badge shows in the HUD and a green ⇪ Warp to map… dropdown appears in the action bar to teleport directly to any of the nine areas. Two ways to enable it:

• Backend: set RG_ADMIN=1 on the server — the world ships pre-unlocked.
• In-game: press Shift + L + A (held together) to toggle admin on/off at any time, even from a normal run. Toggling off restores every lock.

RG_USE_MODEL=0 RG_ADMIN=1 uv run python app/rpg_app.py   # all maps unlocked from boot

NPC dialogue model. RG_USE_DIALOGUE_MODEL=1 enables live, model-written NPC lines from the base MiniCPM-V-4.6 GGUF (text-only llama.cpp load; the first talk loads it in ~10s, then ~3s per line). It is off by default — without it, every NPC uses the deterministic voice tables in src/rune_goblin/story.py, so the game is always playable and quest-critical clues never disappear. Model output is JSON, length-clamped, and its suggested story flags are filtered against an allowlist before they can change the world.

On a MacBook / Apple Silicon machine, prefer running the mounted FastAPI app through uvicorn bound to localhost. This avoids 0.0.0.0 bind issues in restricted environments and uses conservative llama.cpp settings for the local GGUF vision model:

cd app
RG_USE_MODEL=1 \
RG_GGUF_GPU_LAYERS=0 \
RG_GGUF_CTX=2048 \
RG_VISION_MODEL=../models/goblinV1-gguf/gguf/rune-goblin-v46-Q4_K_M.gguf \
RG_VISION_MMPROJ=../models/goblinV1-gguf/gguf/rune-goblin-v46-mmproj-f16.gguf \
PYTHONPATH=../src \
uv run uvicorn rpg_app:app --host 127.0.0.1 --port 7862
# → http://127.0.0.1:7862/play

Controls: WASD / arrows move · 1–9 pick runes (or click the rune deck) · Space cast at whatever you face · E draw a spell (read by goblinV1) · T talk to a faced NPC · J open the journal (quest log, discoveries, story memory) · C clear runes · 🔊 music · ⛶ fullscreen · step into portals to travel. Face an enemy and cast; hit its weakness, and your equipped weapon + class affinity for bonus damage. Locked chests/doors show what runes they need. You weave up to 3 runes per cast until level 3 unlocks 4-rune casts.

The world renders with real Tiny Swords pixel art by Pixel Frog (CC0): grassy islands, knight player, torch goblins, towers and chests. Background music is an original procedural chiptune (Web Audio) that starts on your first click/keypress — toggle it with 🔊. If a sprite is missing the game falls back to emoji, so it always runs.

Deployment note: rpg_app.py mounts FastAPI routes alongside Gradio (gr.mount_gradio_app) and runs under uvicorn. For a Hugging Face Space, use a Docker space (or expose app to uvicorn) rather than the plain Gradio SDK auto-launch.

⚔️ Linear combat game

app/vision_app.py is the simpler 5-room combat game. There are two ways to cast every turn:

• Draw a RuneLang spell on the canvas → the fine-tuned goblinV1 vision model reads your doodle and decides the spell.
• Tap 2–4 runes from the board → instant deterministic rule-engine cast (great for speed, and while the vision model warms up).

# The full game with the downloaded goblinV1 GGUF reading your drawings
RG_USE_MODEL=1 \
RG_VISION_MODEL=models/goblinV1-gguf/gguf/rune-goblin-v46-Q4_K_M.gguf \
RG_VISION_MMPROJ=models/goblinV1-gguf/gguf/rune-goblin-v46-mmproj-f16.gguf \
uv run --extra gguf python app/vision_app.py
# → http://localhost:7861
# The first drawing cast loads the model (~30s on CPU, faster on GPU). Rune-button
# casts are instant. Set RG_USE_MODEL=0 to play purely on the rule engine.

# Minimal legacy rune-button game
uv run python app/app.py                   # → http://localhost:7860

# OR: React frontend + FastAPI backend (rune buttons)
uv run uvicorn api.server:app --port 8000  # terminal 1
cd frontend && npm run dev                 # terminal 2 → http://localhost:5173

Set RG_USE_MODEL=1 (and optionally RG_BASE_MODEL / RG_ADAPTER) to switch the UIs from the rule engine to the fine-tuned model.

For the drawing app, set RG_VISION_MODEL to either a local merged MiniCPM-V Transformers directory or a local .gguf file. For GGUF, also set RG_VISION_MMPROJ to the downloaded projector. The model returns nested visual_reading + spell JSON; the game applies only the validated and clamped spell object.

In the full RPG pipeline, the smaller sketch model should only read the player's drawing and produce metadata such as detected runes, ambiguity, stroke energy and target hints. The stronger asset planner receives validated spell JSON plus player condition, weapon, statuses, power level, enemy state and room/map context, then returns attack/VFX metadata for the renderer.

Notes on the model

• Fine-tuning base: openbmb/MiniCPM-V-4.6 — safetensors, vision-capable, LoRA-trainable. This is what rune-goblin-download and finetune.py use.
• Vision fine-tune: ASHu2/goblinV1 — merged MiniCPM-V model for canvas drawings. The local GGUF files live in models/goblinV1-gguf/gguf/; use app/vision_app.py for the first playable drawing prototype.
• Sketch metadata model: models/goblinV1-gguf/gguf/rune-goblin-v46-Q4_K_M.gguf — fast reader for user sketches and rune metadata.
• Asset planner model: models/MiniCPM-V-4.6-gguf/MiniCPM-V-4_6-Q8_0.gguf — planned higher-quality model for attack type, palette, size, area, path, impact reaction, particle tags and animation timing from validated spell JSON.
• Dialogue / story model: the base (non-fine-tuned) openbmb/MiniCPM-V-4.6 drives NPC dialogue and story progression — loaded as the GGUF models/MiniCPM-V-4.6-gguf/MiniCPM-V-4_6-Q4_K_M.gguf via RG_DIALOGUE_MODEL. It's a quantized GGUF multimodal model and cannot be LoRA-fine-tuned. Download it with --gguf only if you need it.

Renderer rule: do not generate new images during combat or exploration. Use model metadata to drive existing sprites, particles, overlays, CSS/canvas effects, weapon trails, colors, sizes and enemy reactions.

Credits

All art/audio packs are free / CC0. The raw packs live in the git-ignored assets/; a curated subset is baked into app/rpg_static/{sprites,vfx,icons,sfx}/ with a generated manifest.json. scripts/bake_extra_sprites.py re-bakes the extra Tiny Swords assets (terrain tiles, colored buildings, resource piles, skulls, animated flames/TNT goblins) and the snow/stone/dead-tree recolors that give each biome its own ground and props.

• Terrain & buildings: Tiny Swords by Pixel Frog (CC0).
• Creatures (enemies/NPCs): Basic magical animations pack — elementals, golems, witch, treant, fairy, pixie, necromancer, sorceress, druid, wisp.
• Spell VFX: GameFX spritesheets — fireball/cast/burst, ice cast/shatter, poison, tornado, holy explosion, magic barrier, explosions, stars. Cast magic is layered so it grows more elaborate with the spell's tier (rune count, chaos, damage, curse).
• Spell sounds: Retro Magic FX — element spell SFX (fire/ice/electric/ light/dark/earth/water/wind) plus charging/sweeps, played per element on cast.
• Magic circles: Rune Goblin Magic Circles pack — a 100-spell atlas (8-frame 96px rings). On cast, the circle whose runes best match the drawn spell animates under the caster (and the target at higher tiers), growing with the spell tier.
• Rune icons: Mythril Age Icons — one icon per RuneLang glyph (no emoji).
• Music: original procedural chiptune synthesized in-browser via the Web Audio API (no external audio files). 🔊 toggles all music + SFX.