README.md

---
license: apache-2.0
base_model: openbmb/MiniCPM-V-4.6
datasets:
- ASHu2/rune_goblin_visual_dataset
language:
- en
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- minicpmv4_6
- minicpm-v
- vision-language
- multimodal
- image-to-text
- safetensors
- gguf
- lora
- rune-goblin
- runelang
- gradio
- game-ai
- spell-recognition
---

# GoblinV1

**GoblinV1** is a fine-tuned vision-language model for **Rune Goblin**, an AI dungeon crawler where players draw spell glyphs and the model interprets those drawings as magic.

The model is fine-tuned from **OpenBMB MiniCPM-V-4.6** to read hand-drawn **RuneLang** glyphs and produce structured JSON describing detected runes, ambiguity, spell metadata, and visual presentation hints.

It is not a generic chatbot.

It is a tiny cursed spell reader.

> Draw better runes, get stronger magic.  
> Draw cursed doodles, suffer beautifully.

---

## Model Summary

| Field | Value |
|---|---|
| Model name | `ASHu2/goblinV1` |
| Base model | `openbmb/MiniCPM-V-4.6` |
| Model type | Vision-language / image-to-text |
| Architecture | MiniCPM-V 4.6 / `minicpmv4_6` |
| Fine-tuning method | LoRA fine-tuning, merged/exported |
| Primary format | Safetensors |
| Quantized format | GGUF |
| Language | English |
| License | Apache-2.0 |
| Dataset | `ASHu2/rune_goblin_visual_dataset` |
| Primary use case | Hand-drawn rune interpretation for Rune Goblin |

---

## What This Model Does

GoblinV1 reads a player-drawn spell image and returns structured spell interpretation metadata.

The model is trained to identify:

- drawn RuneLang glyphs
- ambiguous or messy rune shapes
- confidence of the visual reading
- spell name and spell type
- colors, shape, motion, particles, and sound tags
- presentation hints for game VFX
- weird, funny, cursed spell flavor consistent with Rune Goblin

The model is designed to act as the **rune reader and spell presentation planner**.

The game engine should still own final combat balance, HP changes, boss rules, inventory state, and durable quest state.

---

## Rune Goblin

Rune Goblin is a Gradio-based AI dungeon crawler where the player draws their own spells.

Instead of selecting “fireball” from a fixed menu, the player draws symbolic glyphs. GoblinV1 reads the glyphs and converts them into spell interpretation JSON.

Example RuneLang meanings:

| Rune | Meaning |
|---|---|
| Flame | burn, danger, passion |
| Leaf | healing, growth, poison |
| Bone | fear, decay, skeletons |
| Spiral | time, confusion, loops |
| Eye | reveal, inspect, prophecy |
| Mirror | reflect, copy, reverse |
| Circle | shield, trap, containment |
| Broken Mark | curse modifier |
| Bell | summon, alarm, attention |
| Coin | trade, greed, sacrifice |

Example combinations:

| Combination | Meaning |
|---|---|
| Flame + Circle | burning shield |
| Spiral + Eye | prophecy / foresight |
| Bone + Dots | skeleton swarm |
| Mirror + Jagged | reflect damage |
| Leaf + Bone | healing with decay risk |
| Broken + Any Rune | stronger effect with cursed side effect |

---

## Expected Input

The model expects an image containing a hand-drawn RuneLang spell.

The image usually contains 1-4 glyphs drawn by the player on a canvas.

The prompt should instruct the model to return JSON only.

Example prompt:

```text
Look at this drawn RuneLang spell. Identify the runes, ambiguity, confidence, and produce spell presentation metadata: name, type, colors, shape, motion, grandeur, particles, and sound tags. Return valid JSON only.
```

---

## Expected Output

GoblinV1 should return JSON in this style:

```json
{
  "visual_reading": {
    "detected_runes": ["flame", "circle"],
    "ambiguous_runes": [],
    "confidence": 0.91,
    "layout": "left_to_right"
  },
  "spell": {
    "spell_name": "Ember Lunchbox Ward",
    "spell_type": "fire_defense",
    "rune_combo": ["flame", "circle"],
    "summary": "A circular flame ward forms around the player.",
    "colors": ["orange", "red", "gold"],
    "shape": "burning circular shield",
    "motion": "slow clockwise rotation with pulsing embers",
    "grandeur": "medium",
    "particles": ["embers", "sparks", "heat shimmer"],
    "sound_tags": ["crackle", "whoosh", "low hum"]
  }
}
```

---

## Important Design Boundary

GoblinV1 is intentionally **not** responsible for final game-state authority.

The model may suggest spell interpretation and presentation metadata, but the game engine should decide:

- final HP changes
- damage numbers
- cooldowns
- boss immunity rules
- quest state
- inventory changes
- progression unlocks
- anti-cheat and validation logic

Recommended architecture:

```text
Player drawing
    ↓
GoblinV1 vision model
    ↓
Rune + spell JSON
    ↓
Game engine validates / clamps / balances
    ↓
Final spell effect is applied
```

This keeps Rune Goblin fun and expressive while preventing model hallucinations from breaking combat balance.

---

## Training Data

GoblinV1 was fine-tuned on `ASHu2/rune_goblin_visual_dataset`, a custom visual instruction dataset for Rune Goblin.

The dataset contains hand-drawn or synthetic RuneLang spell images paired with structured chat-style targets.

Each example teaches the model to:

- read visual glyphs
- map glyphs to RuneLang symbols
- handle messy or ambiguous drawings
- follow RuneLang combination rules
- return valid JSON
- preserve the game’s cursed-comedic tone

The dataset uses image paths and conversational messages containing a system prompt, user prompt, and assistant JSON target.

---

## Training Objective

The objective is not general image captioning.

The model is trained for:

```text
drawn rune image + instruction → structured spell interpretation JSON
```

The model learns the custom RuneLang vocabulary and the relationship between visual glyphs and spell presentation.

---

## Model Formats

This repository includes multiple usable formats.

### Safetensors

Use this for normal Transformers-based inference or deployment.

### GGUF

A quantized GGUF version is included for local inference experiments with llama.cpp-compatible runtimes.

Example GGUF file:

```text
gguf/rune-goblin-v46-Q4_K_M.gguf
```

If using the GGUF model for vision tasks, make sure your runtime supports MiniCPM-V style multimodal inference and loads the required vision/projector files when needed.

---

## Basic Transformers Usage

Install dependencies:

```bash
pip install torch torchvision pillow transformers accelerate
```

Example usage:

```python
from PIL import Image
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

model_id = "ASHu2/goblinV1"

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True
)

model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

image = Image.open("example_rune.png").convert("RGB")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {
                "type": "text",
                "text": (
                    "Look at this drawn RuneLang spell. "
                    "Identify the runes, ambiguity, confidence, and produce spell presentation metadata. "
                    "Return valid JSON only."
                )
            }
        ]
    }
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=False
    )

generated = output_ids[0][inputs["input_ids"].shape[-1]:]
text = processor.decode(generated, skip_special_tokens=True)

print(text)
```

---

## llama.cpp / GGUF Usage

If using a compatible llama.cpp build:

```bash
llama-server -hf ASHu2/goblinV1:Q4_K_M
```

Or run directly:

```bash
llama-cli -hf ASHu2/goblinV1:Q4_K_M
```

For multimodal inference, ensure your client/runtime supports MiniCPM-V and any required vision projector configuration.

---

## Intended Use

GoblinV1 is intended for:

- Rune Goblin gameplay
- hand-drawn spell glyph interpretation
- visual rune recognition
- experimental AI game mechanics
- structured JSON generation from fantasy glyph images
- small-model multimodal game prototypes

---

## Out-of-Scope Use

GoblinV1 is not intended for:

- safety-critical image understanding
- medical, legal, financial, or security decisions
- general OCR benchmarking
- real-world symbol recognition systems
- moderation or surveillance
- authoritative factual QA
- replacing deterministic game rules

---

## Limitations

GoblinV1 may:

- misread very messy drawings
- confuse visually similar glyphs
- produce malformed JSON in some cases
- invent spell details outside the intended schema
- overfit to Rune Goblin-style symbols
- perform poorly on non-RuneLang images
- require game-engine validation before applying effects

Recommended production safeguards:

- validate output JSON
- retry once on invalid output
- clamp all numeric values in the game engine
- reject unknown runes
- keep final state transitions deterministic
- log ambiguous readings for future dataset improvement

---

## Suggested Evaluation

Recommended metrics:

| Metric | Goal |
|---|---|
| Valid JSON rate | >95% |
| Rune detection accuracy | >85% |
| Ambiguity detection quality | Manual review |
| Schema compliance | >95% |
| Unknown rune rejection | High |
| Latency | Playable for Gradio |
| Cursed-fantasy tone consistency | Manual review |

Suggested test cases:

```text
flame + circle
spiral + eye
bone + dots
mirror + jagged
leaf + bone
broken + coin
messy flame vs leaf
partial / incomplete glyph
empty canvas
```

---

## Example Output

```json
{
  "visual_reading": {
    "detected_runes": ["spiral", "eye", "broken_mark"],
    "ambiguous_runes": [
      {
        "candidates": ["spiral", "wave"],
        "reason": "curved repeated stroke could indicate either looping time or water/emotion"
      }
    ],
    "confidence": 0.82,
    "layout": "clustered"
  },
  "spell": {
    "spell_name": "Cursed Foresight Loop",
    "spell_type": "prophecy_curse",
    "rune_combo": ["spiral", "eye", "broken_mark"],
    "summary": "The spell reveals a possible future, then immediately makes it worse.",
    "colors": ["violet", "black", "pale blue"],
    "shape": "floating eye inside a cracked spiral",
    "motion": "spiral contracts inward while the eye flickers",
    "grandeur": "high",
    "particles": ["purple sparks", "black motes", "thin time-rings"],
    "sound_tags": ["whisper", "glass-crack", "reverse-chime"]
  }
}
```

---

## Deployment Notes

GoblinV1 can be used as the model backend for a Gradio game.

Recommended serving layout:

```text
Gradio UI / Canvas
    ↓
Image preprocessing
    ↓
GoblinV1 inference
    ↓
JSON parsing + validation
    ↓
Rune Goblin game engine
    ↓
Updated battle state + animation
```

For GPU deployment, use the Safetensors model with Transformers.

For lightweight/local experiments, use the GGUF export with a compatible llama.cpp runtime.

---

## Related Project

GoblinV1 powers the Rune Goblin dungeon crawler.

Rune Goblin is an AI game where players draw spells, explore maps, fight bosses, trigger cursed outcomes, and unlock stronger effects by drawing clearer runes.

Links:

- Game/demo: https://huggingface.co/spaces/build-small-hackathon/Rune-Goblin
- Model: https://huggingface.co/ASHu2/goblinV1
- Dataset: https://huggingface.co/datasets/ASHu2/rune_goblin_visual_dataset

---

## Citation

If you use this model, please cite:

```bibtex
@misc{goblinv1_2026,
  title        = {GoblinV1: A Fine-Tuned MiniCPM-V Rune Reader for Rune Goblin},
  author       = {Ashutosh Mishra},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {https://huggingface.co/ASHu2/goblinV1}
}
```

Base model:

```bibtex
@misc{minicpmv46_2026,
  title        = {MiniCPM-V 4.6},
  author       = {OpenBMB},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {https://huggingface.co/openbmb/MiniCPM-V-4.6}
}
```

---

## License

This model is released under the Apache-2.0 license.

Please also follow the license and usage terms of the base model `openbmb/MiniCPM-V-4.6`.

---

## Acknowledgements

GoblinV1 is built on top of OpenBMB MiniCPM-V-4.6.

Thanks to the open-source multimodal model community, Hugging Face, Gradio, Modal, llama.cpp, and the cursed little goblin inside every ambiguous doodle.