--- title: Rune Goblin colorFrom: purple colorTo: gray sdk: docker app_port: 7860 pinned: false --- # πŸͺ„ Rune Goblin > Draw bad spells, suffer beautifully. [![Deploy to Hugging Face Space](https://github.com/ASH1998/Rune-Goblin/actions/workflows/deploy-hf-space.yml/badge.svg?branch=master)](https://github.com/ASH1998/Rune-Goblin/actions/workflows/deploy-hf-space.yml) > ### πŸ™ Credits > - πŸ€– **Made with [Codex](https://github.com/codex)** β€” coding and some asset creation (the `BOSS` and `rune_goblin` magic-circle assets are Codex-generated). > - 🧠 **Models by [OpenBMB](https://huggingface.co/openbmb)** β€” [`openbmb/MiniCPM-V-4.6`](https://huggingface.co/openbmb/MiniCPM-V-4.6) (fine-tuned spell engine + base dialogue/story). > - ⚑ **Thanks to [Modal](https://modal.com)** for fine-tuning credits. > > Full attributions, including game art packs, are in [`CREDITS.md`](./CREDITS.md). A tiny dungeon crawler where players draw spells in an invented symbolic language (**RuneLang**) and a fine-tuned [`openbmb/MiniCPM-V-4.6`](https://huggingface.co/openbmb/MiniCPM-V-4.6) acts as the **spell engine** β€” reading glyph combinations and emitting JSON that drives attacks, curses and game-state changes. Runtime visuals are not image-generated; spell metadata recolors, resizes, retargets and animates existing/procedural game assets. See [`rune_goblin_plan.md`](./rune_goblin_plan.md) for the full design doc. ## Architecture ``` Rune buttons (Gradio / React) β†’ serialized rune sequence + game state β†’ fine-tuned openbmb/MiniCPM-V-4.6 + LoRA (rune_goblin.inference) β†’ spell outcome JSON (validated by rune_goblin.schema) β†’ game state engine (rune_goblin.game, clamps HP) β†’ updated UI Drawn canvas (Gradio vision app) β†’ canvas image + game state β†’ goblinV1-gguf Q4_K_M sketch reader β†’ visual_reading / rune metadata (rune_goblin.vision_inference) β†’ fine-tuned RuneLang spell JSON (validated by rune_goblin.schema) β†’ asset / VFX planner (rune_goblin.vfx β€” rule-based today, MiniCPM-V-4.6 later) β†’ attack / VFX metadata (palette, projectile, particles, screen shake) β†’ CSS/canvas renderer + game state engine ``` For the playable MVP the asset planner is a small deterministic function (`rune_goblin.vfx`) instead of a second model: it is instant and never fails, and its metadata shape mirrors the plan so a real `MiniCPM-V-4.6` planner can be swapped in later without touching the renderer. The deterministic rule engine (`rune_goblin.engine`) is both the **dataset oracle** (it generates training targets) and the **runtime fallback** (used until a fine-tuned adapter exists, so the UIs run before training finishes). The **RPG sandbox** (`app/rpg_app.py` + `src/rune_goblin/world.py`) realizes the map-exploration direction: tile-map areas with enemies, NPCs, chests, shrines, powerups, locked doors and a boss. The canvas client owns movement/rendering; `resolve_world_cast` turns drawn/selected runes + the faced target into a validated spell **and** a list of world actions (unlock, loot, defeat, heal, travel-gate…). Python stays the spell engine and balance authority. ## Layout | Path | What | |---|---| | `src/rune_goblin/runelang.py` | 16 runes, grammar, combos, enemies, rooms | | `src/rune_goblin/engine.py` | spell-physics rule engine + HP clamping | | `src/rune_goblin/schema.py` | Pydantic spell schema + JSON repair | | `src/rune_goblin/generate_dataset.py` | synthetic dataset β†’ `data/*.jsonl` | | `src/rune_goblin/finetune.py` | LoRA/QLoRA training (TRL + PEFT) | | `src/rune_goblin/inference.py` | load base + adapter, cast spells | | `src/rune_goblin/vision_inference.py` | load `ASHu2/goblinV1`, read canvas drawings | | `src/rune_goblin/evaluate.py` | game-engine eval metrics | | `src/rune_goblin/vfx.py` | deterministic asset/VFX planner (palette, projectile, particles, shake) | | `src/rune_goblin/game.py` | 5-room dungeon state machine (vision + text/rule cast paths) | | `src/rune_goblin/world.py` | **RPG world**: tile-map areas, entities, and castβ†’world-action resolution | | `app/rpg_app.py` | **Gradio + canvas RPG** β€” free-roaming sandbox (the main game) | | `app/rpg_bridge.py` | FastAPI `/rg/world` + `/rg/cast` bridge for the canvas client | | `app/rpg_static/` | `rpg.js` + `rpg.css` β€” the HTML5 canvas game client | | `app/rpg_static/sprites/` | curated Tiny Swords sprites (CC0) baked for the renderer | | `assets/` | raw Tiny Swords packs (git-ignored; drop the itch download here) | | `app/vision_app.py` | **Gradio** β€” linear 5-room combat game (draw + tap, VFX, rune guide) | | `app/app.py` | minimal Gradio rune-button game (legacy) | | `api/server.py` | FastAPI backend for the React frontend | | `frontend/` | Vite + React custom cursed-dungeon UI | | `notebooks/` | dataset exploration and Modal.com vision fine-tuning notebooks | | `data/` | drop your `.jsonl` training data here | | `models/` | downloaded base model + trained adapters | ## Setup Prerequisites: [`uv`](https://docs.astral.sh/uv/), Node 20+, an `HF_TOKEN` in `.env`. ```bash # 1. Python env (UI + game only) uv sync # 2. Add the heavy ML stack for training / local model inference uv sync --extra train # 2b. Add GGUF runtime support for the local vision model uv sync --extra gguf # 3. Download the fine-tuning base model into models/ uv run rune-goblin-download # add --gguf for the .env serving model # 4. Frontend deps cd frontend && npm install && cd .. ``` ## Workflow ```bash # Generate a synthetic dataset (writes data/rune_spells.jsonl + _val.jsonl) uv run rune-goblin-data --n 5000 # (You can also just drop your own data/*.jsonl β€” same chat format.) # Fine-tune LoRA on your GPU (QLoRA 4-bit by default; 16GB is plenty) uv run rune-goblin-train --data data/rune_spells.jsonl # Evaluate (rule engine, or the model once trained) uv run rune-goblin-eval --n 200 # add --rules-only to skip the model ``` ## Vision fine-tuning on Modal.com For the hand-drawn canvas model, use the Modal.com notebook workflow: 1. Open [`notebooks/MODAL-run-p1Jun2026.ipynb`](./notebooks/MODAL-run-p1Jun2026.ipynb) in Modal Notebooks. 2. Attach a GPU, add your Hugging Face and W&B secrets, and run the notebook top-to-bottom. 3. The notebook prepares the vision JSONL data, fine-tunes `openbmb/MiniCPM-V-4.6` with ms-swift LoRA, evaluates samples, and exports the adapter artifacts. See [`docs/FINETUNE_VISION.md`](./docs/FINETUNE_VISION.md) for the full fine-tuning reference and model-serving notes. ## Run the game ### πŸ—ΊοΈ RPG sandbox (the main game) A free-roaming campaign RPG. You start on a **character-select title screen** and pick one of five goblin heroes (Warrior / Rogue / Poison Rogue / Hunter / Barbarian, art from `assets/Goblin Pack #1`), each with its own HP, courage and rune affinities. Then explore **nine connected areas** β€” Toll Road hub, Mirror Fungus Caverns, Wet Library, Bone Market, Clock Sewer, Gate Approach, the Calendar Beast Arena, plus the **Frostbite Pass** and **Ember Foundry** (a linked ice/forge loop off the hub) β€” fighting enemies, gaining **XP and levels**, buying **weapons** that modify your spells, helping or scaring NPCs, and leaving durable **story flags** that steer one of four endings (Broken / Repaired / Devoured / secret Tollmaster). The final boss has three phases and your hero can **evolve into the Goblin King**. Two models power the game: the fine-tuned `goblinV1` vision model reads your *drawn* runes, and the **base `openbmb/MiniCPM-V-4.6-gguf` model drives interactive NPC dialogue** (press `T` to talk, or cast at an NPC). Movement and rendering run in an HTML5 canvas; casting and dialogue round-trip to Python, which stays the balance authority. The game is served at `/play`. ```bash RG_USE_MODEL=1 \ RG_VISION_MODEL=models/goblinV1-gguf/gguf/rune-goblin-v46-Q4_K_M.gguf \ RG_VISION_MMPROJ=models/goblinV1-gguf/gguf/rune-goblin-v46-mmproj-f16.gguf \ RG_USE_DIALOGUE_MODEL=1 \ RG_DIALOGUE_MODEL=models/MiniCPM-V-4.6-gguf/MiniCPM-V-4_6-Q4_K_M.gguf \ uv run --extra gguf python app/rpg_app.py # β†’ http://localhost:7862 (set RG_USE_MODEL=0 to play purely on the rule engine) ``` **Admin mode.** A fully-unlocked world: every area-to-area portal and locked door is walkable, locked chests open with any cast, and the hero holds the key story items (Calendar Key, Debt Receipt, Thawed Ember…). When it's on, a `πŸ”“ ADMIN` badge shows in the HUD and a green **β‡ͺ Warp to map…** dropdown appears in the action bar to teleport directly to any of the nine areas. Two ways to enable it: - **Backend:** set `RG_ADMIN=1` on the server β€” the world ships pre-unlocked. - **In-game:** press **Shift + L + A** (held together) to toggle admin on/off at any time, even from a normal run. Toggling off restores every lock. ```bash RG_USE_MODEL=0 RG_ADMIN=1 uv run python app/rpg_app.py # all maps unlocked from boot ``` **NPC dialogue model.** `RG_USE_DIALOGUE_MODEL=1` enables live, model-written NPC lines from the *base* MiniCPM-V-4.6 GGUF (text-only `llama.cpp` load; the first talk loads it in ~10s, then ~3s per line). It is **off by default** β€” without it, every NPC uses the deterministic voice tables in `src/rune_goblin/story.py`, so the game is always playable and quest-critical clues never disappear. Model output is JSON, length-clamped, and its suggested story flags are filtered against an allowlist before they can change the world. On a MacBook / Apple Silicon machine, prefer running the mounted FastAPI app through `uvicorn` bound to localhost. This avoids `0.0.0.0` bind issues in restricted environments and uses conservative llama.cpp settings for the local GGUF vision model: ```bash cd app RG_USE_MODEL=1 \ RG_GGUF_GPU_LAYERS=0 \ RG_GGUF_CTX=2048 \ RG_VISION_MODEL=../models/goblinV1-gguf/gguf/rune-goblin-v46-Q4_K_M.gguf \ RG_VISION_MMPROJ=../models/goblinV1-gguf/gguf/rune-goblin-v46-mmproj-f16.gguf \ PYTHONPATH=../src \ uv run uvicorn rpg_app:app --host 127.0.0.1 --port 7862 # β†’ http://127.0.0.1:7862/play ``` **Controls:** `WASD` / arrows move Β· `1–9` pick runes (or click the rune deck) Β· `Space` cast at whatever you face Β· `E` draw a spell (read by goblinV1) Β· `T` talk to a faced NPC Β· `J` open the journal (quest log, discoveries, story memory) Β· `C` clear runes Β· `πŸ”Š` music Β· `β›Ά` fullscreen Β· step into portals to travel. Face an enemy and cast; hit its weakness, and your equipped weapon + class affinity for bonus damage. Locked chests/doors show what runes they need. You weave up to 3 runes per cast until level 3 unlocks 4-rune casts. The world renders with real **[Tiny Swords](https://pixelfrog-assets.itch.io/tiny-swords)** pixel art by **Pixel Frog** (CC0): grassy islands, knight player, torch goblins, towers and chests. Background music is an original procedural chiptune (Web Audio) that starts on your first click/keypress β€” toggle it with `πŸ”Š`. If a sprite is missing the game falls back to emoji, so it always runs. > Deployment note: `rpg_app.py` mounts FastAPI routes alongside Gradio > (`gr.mount_gradio_app`) and runs under `uvicorn`. For a Hugging Face Space, use > a Docker space (or expose `app` to uvicorn) rather than the plain Gradio SDK > auto-launch. ### βš”οΈ Linear combat game `app/vision_app.py` is the simpler 5-room combat game. There are two ways to cast every turn: - **Draw** a RuneLang spell on the canvas β†’ the fine-tuned `goblinV1` vision model reads your doodle and decides the spell. - **Tap 2–4 runes** from the board β†’ instant deterministic rule-engine cast (great for speed, and while the vision model warms up). ```bash # The full game with the downloaded goblinV1 GGUF reading your drawings RG_USE_MODEL=1 \ RG_VISION_MODEL=models/goblinV1-gguf/gguf/rune-goblin-v46-Q4_K_M.gguf \ RG_VISION_MMPROJ=models/goblinV1-gguf/gguf/rune-goblin-v46-mmproj-f16.gguf \ uv run --extra gguf python app/vision_app.py # β†’ http://localhost:7861 # The first drawing cast loads the model (~30s on CPU, faster on GPU). Rune-button # casts are instant. Set RG_USE_MODEL=0 to play purely on the rule engine. # Minimal legacy rune-button game uv run python app/app.py # β†’ http://localhost:7860 # OR: React frontend + FastAPI backend (rune buttons) uv run uvicorn api.server:app --port 8000 # terminal 1 cd frontend && npm run dev # terminal 2 β†’ http://localhost:5173 ``` Set `RG_USE_MODEL=1` (and optionally `RG_BASE_MODEL` / `RG_ADAPTER`) to switch the UIs from the rule engine to the fine-tuned model. For the drawing app, set `RG_VISION_MODEL` to either a local merged MiniCPM-V Transformers directory or a local `.gguf` file. For GGUF, also set `RG_VISION_MMPROJ` to the downloaded projector. The model returns nested `visual_reading` + `spell` JSON; the game applies only the validated and clamped `spell` object. In the full RPG pipeline, the smaller sketch model should only read the player's drawing and produce metadata such as detected runes, ambiguity, stroke energy and target hints. The stronger asset planner receives validated spell JSON plus player condition, weapon, statuses, power level, enemy state and room/map context, then returns attack/VFX metadata for the renderer. ## Notes on the model - **Fine-tuning base**: `openbmb/MiniCPM-V-4.6` β€” safetensors, vision-capable, LoRA-trainable. This is what `rune-goblin-download` and `finetune.py` use. - **Vision fine-tune**: `ASHu2/goblinV1` β€” merged MiniCPM-V model for canvas drawings. The local GGUF files live in `models/goblinV1-gguf/gguf/`; use `app/vision_app.py` for the first playable drawing prototype. - **Sketch metadata model**: `models/goblinV1-gguf/gguf/rune-goblin-v46-Q4_K_M.gguf` β€” fast reader for user sketches and rune metadata. - **Asset planner model**: `models/MiniCPM-V-4.6-gguf/MiniCPM-V-4_6-Q8_0.gguf` β€” planned higher-quality model for attack type, palette, size, area, path, impact reaction, particle tags and animation timing from validated spell JSON. - **Dialogue / story model**: the base (non-fine-tuned) `openbmb/MiniCPM-V-4.6` drives NPC dialogue and story progression β€” loaded as the GGUF `models/MiniCPM-V-4.6-gguf/MiniCPM-V-4_6-Q4_K_M.gguf` via `RG_DIALOGUE_MODEL`. It's a **quantized GGUF multimodal** model and **cannot** be LoRA-fine-tuned. Download it with `--gguf` only if you need it. Renderer rule: do not generate new images during combat or exploration. Use model metadata to drive existing sprites, particles, overlays, CSS/canvas effects, weapon trails, colors, sizes and enemy reactions. ## Credits All art/audio packs are free / CC0. The raw packs live in the git-ignored `assets/`; a curated subset is baked into `app/rpg_static/{sprites,vfx,icons,sfx}/` with a generated `manifest.json`. `scripts/bake_extra_sprites.py` re-bakes the extra Tiny Swords assets (terrain tiles, colored buildings, resource piles, skulls, animated flames/TNT goblins) and the snow/stone/dead-tree recolors that give each biome its own ground and props. - **Terrain & buildings**: [Tiny Swords](https://pixelfrog-assets.itch.io/tiny-swords) by **Pixel Frog** (CC0). - **Creatures (enemies/NPCs)**: Basic magical animations pack β€” elementals, golems, witch, treant, fairy, pixie, necromancer, sorceress, druid, wisp. - **Spell VFX**: GameFX spritesheets β€” fireball/cast/burst, ice cast/shatter, poison, tornado, holy explosion, magic barrier, explosions, stars. Cast magic is layered so it grows more elaborate with the spell's tier (rune count, chaos, damage, curse). - **Spell sounds**: Retro Magic FX β€” element spell SFX (fire/ice/electric/ light/dark/earth/water/wind) plus charging/sweeps, played per element on cast. - **Magic circles**: Rune Goblin Magic Circles pack β€” a 100-spell atlas (8-frame 96px rings). On cast, the circle whose runes best match the drawn spell animates under the caster (and the target at higher tiers), growing with the spell tier. - **Rune icons**: Mythril Age Icons β€” one icon per RuneLang glyph (no emoji). - **Music**: original procedural chiptune synthesized in-browser via the Web Audio API (no external audio files). `πŸ”Š` toggles all music + SFX.