Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
execute this project plan

--- Attached Context ---

📄 @file:`QUICKSTART.md` (551 tokens)
```markdown
# Quick Start

## 1. Run the Fine-Tuned Model

### MLX (recommended for inference):
```bash
mlx_lm.generate --model ./fused_model --prompt "Your Unity task..."
```

### Ollama:
Create Modelfile in this directory:
```dockerfile
FROM ./gguf/unity-agent-Q6_K.gguf
TEMPLATE "{{ .Prompt }}"
PARAMETER temperature 0.3
PARAMETER num_ctx 4096
```

Then:
```bash
ollama create unity-agent -f Modelfile
ollama serve
```

### LM Studio:
- Open LM Studio → "My Models" → "Add Model"
- Select ./gguf/unity-agent-Q6_K.gguf
- Start server on port 1234

## 2. Connect to Unity Editor

### Option A: Unity-MCP CLI (recommended)
```bash
npx -y @aigamedeveloper/mcp-server
# Connects Claude Desktop, Cursor, or VS Code to Unity Editor
```

### Option B: Custom Editor Window
- Copy `mcp_config/UnityAgentWindow.cs` to your project's `Assets/Editor/`
- Window → Unity Agent → Chat

### Option C: Python client
```bash
python3 mcp_config/unity_mcp_client.py --list
python3 mcp_config/unity_mcp_client.py --tool gameobject-create \
  --args '{"name":"Cube","primitiveType":"Cube"}'
```

## 3. Example Workflows

**Create a scene from scratch:**
> "Create a medieval village with 5 houses, a central fountain, and a player character"

**Fix bugs interactively:**
> "Read the console logs and fix the NullReferenceException in PlayerHealth.cs"

**Profile and optimize:**
> "Profile the game, find the bottleneck, and suggest optimizations"

**Design a system:**
> "Create a ScriptableObject-based inventory system with item pickup, UI display, and save/load"

## 4. Dataset Pipeline (full run)

```bash
# Generate custom datasets
python3 01_build_unity_mcp_dataset.py
python3 02_build_unity_design_patterns_dataset.py

# Download external datasets
python3 05_download_and_merge_datasets.py
python3 06_extract_unity_repos.py
python3 07_integrate_new_datasets.py

# Train + export
python3 03_train_and_export_unity_agent.py
```

## 5. Fine-Tuning Stages

```bash
# Prepare only (merge datasets)
python3 03_train_and_export_unity_agent.py --stage prepare

# Train only (QLoRA)
python3 03_train_and_export_unity_agent.py --stage train

# Fuse adapters and export GGUF
python3 03_train_and_export_unity_agent.py --stage export
```

```

📄 @file:`README.md` (1740 tokens)
```markdown
# Unity Agent Developer — Model Training Pipeline

Build and deploy a Unity game development agent model on Apple Silicon (M5 Pro).

## Quick Start

```bash
# 1. Generate custom datasets
python3 01_build_unity_mcp_dataset.py
python3 02_build_unity_design_patterns_dataset.py

# 2. Download + merge everything (internet needed)
python3 05_download_and_merge_datasets.py       # vishnuOI + function-calling
python3 06_extract_unity_repos.py                # ML-Agents, ECS, uGUI, netcode
python3 07_integrate_new_datasets.py             # Erocal Q&A + Mix & Jam mechanics

# 3. Fine-tune (MLX QLoRA on Apple Silicon)
python3 03_train_and_export_unity_agent.py

# 4. Export to GGUF
python3 03_train_and_export_unity_agent.py --stage export

# 5. Run
ollama create unity-agent -f Modelfile
ollama run unity-agent
```

## Components

| # | Script | What It Does | Output |
|---|---|---|---|
| 01 | `01_build_unity_mcp_dataset.py` | Generates ~128 MCP tool-calling examples from real game-dev tasks | `datasets/unity_mcp_tool_calls_*.jsonl` |
| 02 | `02_build_unity_design_patterns_dataset.py` | Generates ~75 design pattern + bug-fix examples | `datasets/unity_design_patterns_*.jsonl` |
| 03 | `03_train_and_export_unity_agent.py` | MLX QLoRA train → fuse → GGUF export → eval | `adapters/`, `fused_model/`, `gguf/` |
| 04 | `04_setup_unity_mcp_bridge.py` | Sets up Unity-MCP bridge, configs for Claude/Cursor/VS Code | `mcp_config/` |
| 05 | `05_download_and_merge_datasets.py` | Downloads `vishnuOI/unity-dev-instructions` (46k) + `hermes-function-calling-v1` (2k), merges everything | `datasets/train.jsonl` |
| 06 | `06_extract_unity_repos.py` | Extracts C# code from Unity repos: ML-Agents (29), ECS (10), uGUI (10), Netcode (5) | `datasets/unity_repos_*.jsonl` |
| 07 | `07_integrate_new_datasets.py` | Downloads + integrates `Erocal/Unity_ResolvedQuestions` (400 real Q&A) and `MixAndJam/GameMechanics` (32 game mechanic C# codes) | Rebuilds `datasets/{train,valid,test}.jsonl` |

## Dataset Composition (after all merges)

| Source | Rows | Coverage |
|---|---|---|
| `vishnuOI/unity-dev-instructions` | ~42k (after dedup) | Unity API docs, Stack Overflow, GitHub code — scripting, rendering, physics, editor, XR, UI, animation, networking, math, performance, input |
| `NousResearch/hermes-function-calling-v1` | ~1.8k | Generic tool-calling / JSON-mode structured outputs |
| `Erocal/Unity_ResolvedQuestions` | ~400 | Real resolved Q&A — physics, ECS, URP/HDRP, UI Toolkit, editor tooling, platform issues, animation rigging |
| `AmnaHassan/MixAndJam-GameMechanics` | ~32 | Full game mechanic C# implementations: Celeste movement, Splatoon ink, Arkham combat, God of War axe throw, Balatro feel, AC dialogue, Luigi's Mansion ghost capture, Mario Galaxy launch stars, Pikmin gameplay |
| ML-Agents example agents | ~29 | RL agent patterns (Crawler, Walker, FoodCollector, Soccer, Pyramids, WallJump, Worm, Hallway, 3DBall, PushBlock, GridWorld, Sorter) |
| ML-Agents core + sensors | ~5 | Agent.cs lifecycle, ray perception, camera sensor, vector sensor, grid sensor |
| ECS/DOTS samples | ~10 | Jobs, SystemBase, ISystem, IComponentData, IAspect, Baker patterns |
| uGUI | ~10 | Graphic, Image, Text, Button, Slider, Toggle, ScrollRect, Dropdown, InputField, LayoutGroup |
| Netcode for GameObjects | ~5 | NetworkBehaviour, NetworkVariable, RPC, NetworkObject patterns |
| Unity MCP tool calls (custom) | ~128 | Agentic Unity tool use: scene setup, scripting, prefabs, debugging, profiling, testing, shaders, reflection |
| Design patterns (custom) | ~75 | ScriptableObject architecture, Object Pooling, Singleton, State Machine, Observer, Strategy, Command, Factory, Flyweight, Save/Load, ServiceLocator, EventBus |
| **Total (deduped)** | **~45k** | |

### Datasets considered and rejected

| Source | Why skipped |
|---|---|
| `ibranze/codellama_unity3d_v1/v2` | 535 + 2k rows, wrong format, overlaps with vishnuOI |
| `ZorraZabb/code25pile75_shuffle_filtered_unity3d_dedup` | 1M rows of noise — substring "unity3d" filter catches Python, JS, news articles, medical papers |
| `supergoose/buzz_sources_244_unity3d-asset` | 101 rows of git diff YAML for .meta files |
| `robertzeng/unity3ddoc` | 113k rows but shallow/trivial responses (5-15 word generic definitions) |
| `DLMveloper/DLM_DataSet` | 3.6M rows but only ~3,570 "Unity"-tagged, all trivial (variable declarations, basic array ops) |
| `Saibalaji25/unity-csharp-scripts` | Requires login + T&C agreement, unknown quality, 11 downloads/mo |

## Training Configuration (M5 Pro 64GB)

```
Model:      Qwen2.5-Coder-32B-Instruct (4-bit QLoRA)
Batch size: 1
Layers:     16/64 (25% LoRA)
LR:         1e-4
Max seq:    4096
Iters:      2000
Peak mem:   ~25GB
Duration:   ~45-60 min
```

## GGUF Output Quantization

| Type | Size | Best For |
|---|---|---|
| Q4_K_M | ~20 GB | 24GB+ systems |
| Q5_K_M | ~23 GB | 32GB+ systems |
| **Q6_K** | **~27 GB** | **M5 Pro 64GB (recommended)** |
| Q8_0 | ~34 GB | 64GB+ systems (max quality) |

## Unity-MCP Bridge

Run the bridge setup to configure connections to Claude Desktop, Cursor, VS Code, or the included Python client:

```bash
python3 04_setup_unity_mcp_bridge.py
```

Then connect any MCP-compatible AI client (Claude Desktop, Cursor, VS Code with Cline/Copilot, Codex) to the Unity Editor.

## Project Structure

```
unity-agent-training/
├── 01_build_unity_mcp_dataset.py            # MCP tool-calling dataset
├── 02_build_unity_design_patterns_dataset.py  # Design patterns dataset
├── 03_train_and_export_unity_agent.py        # MLX training + GGUF export
├── 04_setup_unity_mcp_bridge.py              # Unity-MCP integration
├── 05_download_and_merge_datasets.py         # HF dataset download + merge
├── 06_extract_unity_repos.py                 # Unity org repo C# extraction
├── 07_integrate_new_datasets.py              # Newly discovered datasets
├── Modelfile                                 # Ollama model definition
├── QUICKSTART.md                             # Quick start guide
├── datasets/
│   ├── unity_mcp_tool_calls_{train,valid,test}.jsonl
│   ├── unity_design_patterns_{train,valid,test}.jsonl
│   ├── unity_api_instructions.jsonl          # (downloaded)
│   ├── unity_repos_{train,valid,test}.jsonl  # (from repo extraction)
│   ├── function_calling_full.jsonl           # (downloaded)
│   ├── train.jsonl                           # Merged MLX-ready
│   ├── valid.jsonl                           # Merged MLX-ready
│   └── test.jsonl                            # Merged MLX-ready
├── adapters/                                 # LoRA adapters (after training)
├── fused_model/                              # Fused MLX model
├── gguf/                                     # GGUF quantized models
└── mcp_config/                               # Bridge configuration
    ├── claude_desktop_config.json
    ├── vscode_mcp.json
    ├── unity_mcp_server_config.json
    ├── unity_mcp_client.py
    └── UnityAgentWindow.cs
```

```<|im_end|>


Prompt:
<|im_start|>system
You are in Data Analyst profile. Prioritize data loading, Python analysis, and charting.
<|im_end|>
<|im_start|>user
find me best Llama 3.3 70B custom fine-tuned model for coding<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Research the best Hermes Agent plugin configuration for an 'image-generator' profile focused on AI image generation, image editing, design workflows, and visual content creation. Consider which plugins (model providers, web search, browser, memory, tools) are most relevant for an image generation agent. Focus on practical recommendations: which plugins to enable, which to skip, and why. Check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
I need to process this. Please help.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
I need to process proc_0a497b8b5935 exited (exit code -15,. Please help.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
[IMPORTANT: Background process proc_0a497b8b5935 exited (exit code -15, SIGTERM).
Command: unset PYTHONPATH; /Users/jacobeen/.hermes/hermes-agent/venv/bin/python3 03_train_and_export_unity_agent.py train 2>&1
Output:
4<00:26,  1.25s/it]
Calculating loss...:  20%|██        | 5/25 [00:04<00:21,  1.08s/it]
Calculating loss...:  24%|██▍       | 6/25 [00:05<00:16,  1.18it/s]
Calculating loss...:  28%|██▊       | 7/25 [00:06<00:16,  1.07it/s]
Calculating loss...:  32%|███▏      | 8/25 [00:07<00:15,  1.07it/s]
Calculating loss...:  36%|███▌      | 9/25 [00:08<00:17,  1.10s/it]
Calculating loss...:  40%|████      | 10/25 [00:10<00:19,  1.27s/it]
Calculating loss...:  44%|████▍     | 11/25 [00:11<00:17,  1.28s/it]
Calculating loss...:  48%|████▊     | 12/25 [00:12<00:14,  1.13s/it]
Calculating loss...:  52%|█████▏    | 13/25 [00:14<00:14,  1.24s/it]
Calculating loss...:  56%|█████▌    | 14/25 [00:14<00:10,  1.02it/s]
Calculating loss...:  60%|██████    | 15/25 [00:14<00:08,  1.24it/s]
Calculating loss...:  64%|██████▍   | 16/25 [00:16<00:09,  1.00s/it]
Calculating loss...:  68%|██████▊   | 17/25 [00:16<00:07,  1.13it/s]
Calculating loss...:  72%|███████▏  | 18/25 [00:17<00:06,  1.11it/s]
Calculating loss...:  76%|███████▌  | 19/25 [00:18<00:04,  1.25it/s]
Calculating loss...:  80%|████████  | 20/25 [00:18<00:03,  1.46it/s]
Calculating loss...:  84%|████████▍ | 21/25 [00:19<00:03,  1.33it/s]
Calculating loss...:  88%|████████▊ | 22/25 [00:20<00:02,  1.24it/s]
Calculating loss...:  92%|█████████▏| 23/25 [00:21<00:01,  1.26it/s]
Calculating loss...:  96%|█████████▌| 24/25 [00:22<00:00,  1.27it/s]
Calculating loss...: 100%|██████████| 25/25 [00:24<00:00,  1.10s/it]
Calculating loss...: 100%|██████████| 25/25 [00:24<00:00,  1.04it/s]
Iter 100: Val loss nan, Val took 24.072s
Iter 100: Train loss nan, Learning Rate 1.000e-04, It/sec 0.478, Tokens/sec 10225505.663, Trained Tokens 1068717925, Peak mem 65.672 GB
/Users/jacobeen/.local/share/uv/python/cpython-3.11.15-macos-aarch64-none/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
]<|im_end|>


Prompt:
<|im_start|>system
You are in General Assistant profile. Use the most efficient tool for each request.
<|im_end|>
<|im_start|>user
[IMPORTANT: Background process proc_0a497b8b5935 exited (exit code -15, SIGTERM).
Command: unset PYTHONPATH; /Users/jacobeen/.hermes/hermes-agent/venv/bin/python3 03_train_and_export_unity_agent.py train 2>&1
Output:
4<00:26,  1.25s/it]
Calculating loss...:  20%|██        | 5/25 [00:04<00:21,  1.08s/it]
Calculating loss...:  24%|██▍       | 6/25 [00:05<00:16,  1.18it/s]
Calculating loss...:  28%|██▊       | 7/25 [00:06<00:16,  1.07it/s]
Calculating loss...:  32%|███▏      | 8/25 [00:07<00:15,  1.07it/s]
Calculating loss...:  36%|███▌      | 9/25 [00:08<00:17,  1.10s/it]
Calculating loss...:  40%|████      | 10/25 [00:10<00:19,  1.27s/it]
Calculating loss...:  44%|████▍     | 11/25 [00:11<00:17,  1.28s/it]
Calculating loss...:  48%|████▊     | 12/25 [00:12<00:14,  1.13s/it]
Calculating loss...:  52%|█████▏    | 13/25 [00:14<00:14,  1.24s/it]
Calculating loss...:  56%|█████▌    | 14/25 [00:14<00:10,  1.02it/s]
Calculating loss...:  60%|██████    | 15/25 [00:14<00:08,  1.24it/s]
Calculating loss...:  64%|██████▍   | 16/25 [00:16<00:09,  1.00s/it]
Calculating loss...:  68%|██████▊   | 17/25 [00:16<00:07,  1.13it/s]
Calculating loss...:  72%|███████▏  | 18/25 [00:17<00:06,  1.11it/s]
Calculating loss...:  76%|███████▌  | 19/25 [00:18<00:04,  1.25it/s]
Calculating loss...:  80%|████████  | 20/25 [00:18<00:03,  1.46it/s]
Calculating loss...:  84%|████████▍ | 21/25 [00:19<00:03,  1.33it/s]
Calculating loss...:  88%|████████▊ | 22/25 [00:20<00:02,  1.24it/s]
Calculating loss...:  92%|█████████▏| 23/25 [00:21<00:01,  1.26it/s]
Calculating loss...:  96%|█████████▌| 24/25 [00:22<00:00,  1.27it/s]
Calculating loss...: 100%|██████████| 25/25 [00:24<00:00,  1.10s/it]
Calculating loss...: 100%|██████████| 25/25 [00:24<00:00,  1.04it/s]
Iter 100: Val loss nan, Val took 24.072s
Iter 100: Train loss nan, Learning Rate 1.000e-04, It/sec 0.478, Tokens/sec 10225505.663, Trained Tokens 1068717925, Peak mem 65.672 GB
/Users/jacobeen/.local/share/uv/python/cpython-3.11.15-macos-aarch64-none/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
]<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
can you make sure that every single profile is utilizing our Obsidian.md, Hindsight and othr plugins?

[System: The active model for this chat has changed to Ornith-1.0-35B-MTPLX via provider omlx-mbp. From this point forward, use this runtime metadata when answering questions about what model/provider is active.]

can you make sure that every single profile is utilizing our Obsidian.md, Hindsight and othr plugins?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Can you process this?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
i need to know if you are making a GGUF model or MLX model?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Research the best Hermes Agent plugin configuration for a 'game-dev' profile focused on Unity game development, 3D graphics, game logic, shaders, and game engine workflows. Consider which plugins (model providers, web search, browser, memory, tools) are most relevant for a game developer. Focus on practical recommendations: which plugins to enable, which to skip, and why. Check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
oh okey so we want to MAX out the quality at 2000<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Research the best Hermes Agent plugin configuration for a 'cybersec' profile focused on security auditing, penetration testing, vulnerability research, and security analysis. Consider which plugins (model providers, web search, browser, memory, tools) are most relevant for a security-focused agent. Focus on practical recommendations: which plugins to enable, which to skip, and why. Check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.<|im_end|>


Prompt:
<|im_start|>system
You are in System Admin profile. Prioritize terminal commands, process management, and system tools.
<|im_end|>
<|im_start|>user
progress?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
[IMPORTANT: Background process proc_0a497b8b5935 exited (exit code -15, SIGTERM).
Command: unset PYTHONPATH; /Users/jacobeen/.hermes/hermes-agent/venv/bin/python3 03_train_and_export_unity_agent.py train 2>&1
Output:
4<00:26,  1.25s/it]
Calculating loss...:  20%|██        | 5/25 [00:04<00:21,  1.08s/it]
Calculating loss...:  24%|██▍       | 6/25 [00:05<00:16,  1.18it/s]
Calculating loss...:  28%|██▊       | 7/25 [00:06<00:16,  1.07it/s]
Calculating loss...:  32%|███▏      | 8/25 [00:07<00:15,  1.07it/s]
Calculating loss...:  36%|███▌      | 9/25 [00:08<00:17,  1.10s/it]
Calculating loss...:  40%|████      | 10/25 [00:10<00:19,  1.27s/it]
Calculating loss...:  44%|████▍     | 11/25 [00:11<00:17,  1.28s/it]
Calculating loss...:  48%|████▊     | 12/25 [00:12<00:14,  1.13s/it]
Calculating loss...:  52%|█████▏    | 13/25 [00:14<00:14,  1.24s/it]
Calculating loss...:  56%|█████▌    | 14/25 [00:14<00:10,  1.02it/s]
Calculating loss...:  60%|██████    | 15/25 [00:14<00:08,  1.24it/s]
Calculating loss...:  64%|██████▍   | 16/25 [00:16<00:09,  1.00s/it]
Calculating loss...:  68%|██████▊   | 17/25 [00:16<00:07,  1.13it/s]
Calculating loss...:  72%|███████▏  | 18/25 [00:17<00:06,  1.11it/s]
Calculating loss...:  76%|███████▌  | 19/25 [00:18<00:04,  1.25it/s]
Calculating loss...:  80%|████████  | 20/25 [00:18<00:03,  1.46it/s]
Calculating loss...:  84%|████████▍ | 21/25 [00:19<00:03,  1.33it/s]
Calculating loss...:  88%|████████▊ | 22/25 [00:20<00:02,  1.24it/s]
Calculating loss...:  92%|█████████▏| 23/25 [00:21<00:01,  1.26it/s]
Calculating loss...:  96%|█████████▌| 24/25 [00:22<00:00,  1.27it/s]
Calculating loss...: 100%|██████████| 25/25 [00:24<00:00,  1.10s/it]
Calculating loss...: 100%|██████████| 25/25 [00:24<00:00,  1.04it/s]
Iter 100: Val loss nan, Val took 24.072s
Iter 100: Train loss nan, Learning Rate 1.000e-04, It/sec 0.478, Tokens/sec 10225505.663, Trained Tokens 1068717925, Peak mem 65.672 GB
/Users/jacobeen/.local/share/uv/python/cpython-3.11.15-macos-aarch64-none/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
]<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
i need to know if you are making a GGUF model or MLX model?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Can you check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
execute this project plan

--- Attached Context ---

📄 @file:`QUICKSTART.md` (551 tokens)
```markdown
# Quick Start

## 1. Run the Fine-Tuned Model

### MLX (recommended for inference):
```bash
mlx_lm.generate --model ./fused_model --prompt "Your Unity task..."
```

### Ollama:
Create Modelfile in this directory:
```dockerfile
FROM ./gguf/unity-agent-Q6_K.gguf
TEMPLATE "{{ .Prompt }}"
PARAMETER temperature 0.3
PARAMETER num_ctx 4096
```

Then:
```bash
ollama create unity-agent -f Modelfile
ollama serve
```

### LM Studio:
- Open LM Studio → "My Models" → "Add Model"
- Select ./gguf/unity-agent-Q6_K.gguf
- Start server on port 1234

## 2. Connect to Unity Editor

### Option A: Unity-MCP CLI (recommended)
```bash
npx -y @aigamedeveloper/mcp-server
# Connects Claude Desktop, Cursor, or VS Code to Unity Editor
```

### Option B: Custom Editor Window
- Copy `mcp_config/UnityAgentWindow.cs` to your project's `Assets/Editor/`
- Window → Unity Agent → Chat

### Option C: Python client
```bash
python3 mcp_config/unity_mcp_client.py --list
python3 mcp_config/unity_mcp_client.py --tool gameobject-create \
  --args '{"name":"Cube","primitiveType":"Cube"}'
```

## 3. Example Workflows

**Create a scene from scratch:**
> "Create a medieval village with 5 houses, a central fountain, and a player character"

**Fix bugs interactively:**
> "Read the console logs and fix the NullReferenceException in PlayerHealth.cs"

**Profile and optimize:**
> "Profile the game, find the bottleneck, and suggest optimizations"

**Design a system:**
> "Create a ScriptableObject-based inventory system with item pickup, UI display, and save/load"

## 4. Dataset Pipeline (full run)

```bash
# Generate custom datasets
python3 01_build_unity_mcp_dataset.py
python3 02_build_unity_design_patterns_dataset.py

# Download external datasets
python3 05_download_and_merge_datasets.py
python3 06_extract_unity_repos.py
python3 07_integrate_new_datasets.py

# Train + export
python3 03_train_and_export_unity_agent.py
```

## 5. Fine-Tuning Stages

```bash
# Prepare only (merge datasets)
python3 03_train_and_export_unity_agent.py --stage prepare

# Train only (QLoRA)
python3 03_train_and_export_unity_agent.py --stage train

# Fuse adapters and export GGUF
python3 03_train_and_export_unity_agent.py --stage export
```

```

📄 @file:`README.md` (1740 tokens)
```markdown
# Unity Agent Developer — Model Training Pipeline

Build and deploy a Unity game development agent model on Apple Silicon (M5 Pro).

## Quick Start

```bash
# 1. Generate custom datasets
python3 01_build_unity_mcp_dataset.py
python3 02_build_unity_design_patterns_dataset.py

# 2. Download + merge everything (internet needed)
python3 05_download_and_merge_datasets.py       # vishnuOI + function-calling
python3 06_extract_unity_repos.py                # ML-Agents, ECS, uGUI, netcode
python3 07_integrate_new_datasets.py             # Erocal Q&A + Mix & Jam mechanics

# 3. Fine-tune (MLX QLoRA on Apple Silicon)
python3 03_train_and_export_unity_agent.py

# 4. Export to GGUF
python3 03_train_and_export_unity_agent.py --stage export

# 5. Run
ollama create unity-agent -f Modelfile
ollama run unity-agent
```

## Components

| # | Script | What It Does | Output |
|---|---|---|---|
| 01 | `01_build_unity_mcp_dataset.py` | Generates ~128 MCP tool-calling examples from real game-dev tasks | `datasets/unity_mcp_tool_calls_*.jsonl` |
| 02 | `02_build_unity_design_patterns_dataset.py` | Generates ~75 design pattern + bug-fix examples | `datasets/unity_design_patterns_*.jsonl` |
| 03 | `03_train_and_export_unity_agent.py` | MLX QLoRA train → fuse → GGUF export → eval | `adapters/`, `fused_model/`, `gguf/` |
| 04 | `04_setup_unity_mcp_bridge.py` | Sets up Unity-MCP bridge, configs for Claude/Cursor/VS Code | `mcp_config/` |
| 05 | `05_download_and_merge_datasets.py` | Downloads `vishnuOI/unity-dev-instructions` (46k) + `hermes-function-calling-v1` (2k), merges everything | `datasets/train.jsonl` |
| 06 | `06_extract_unity_repos.py` | Extracts C# code from Unity repos: ML-Agents (29), ECS (10), uGUI (10), Netcode (5) | `datasets/unity_repos_*.jsonl` |
| 07 | `07_integrate_new_datasets.py` | Downloads + integrates `Erocal/Unity_ResolvedQuestions` (400 real Q&A) and `MixAndJam/GameMechanics` (32 game mechanic C# codes) | Rebuilds `datasets/{train,valid,test}.jsonl` |

## Dataset Composition (after all merges)

| Source | Rows | Coverage |
|---|---|---|
| `vishnuOI/unity-dev-instructions` | ~42k (after dedup) | Unity API docs, Stack Overflow, GitHub code — scripting, rendering, physics, editor, XR, UI, animation, networking, math, performance, input |
| `NousResearch/hermes-function-calling-v1` | ~1.8k | Generic tool-calling / JSON-mode structured outputs |
| `Erocal/Unity_ResolvedQuestions` | ~400 | Real resolved Q&A — physics, ECS, URP/HDRP, UI Toolkit, editor tooling, platform issues, animation rigging |
| `AmnaHassan/MixAndJam-GameMechanics` | ~32 | Full game mechanic C# implementations: Celeste movement, Splatoon ink, Arkham combat, God of War axe throw, Balatro feel, AC dialogue, Luigi's Mansion ghost capture, Mario Galaxy launch stars, Pikmin gameplay |
| ML-Agents example agents | ~29 | RL agent patterns (Crawler, Walker, FoodCollector, Soccer, Pyramids, WallJump, Worm, Hallway, 3DBall, PushBlock, GridWorld, Sorter) |
| ML-Agents core + sensors | ~5 | Agent.cs lifecycle, ray perception, camera sensor, vector sensor, grid sensor |
| ECS/DOTS samples | ~10 | Jobs, SystemBase, ISystem, IComponentData, IAspect, Baker patterns |
| uGUI | ~10 | Graphic, Image, Text, Button, Slider, Toggle, ScrollRect, Dropdown, InputField, LayoutGroup |
| Netcode for GameObjects | ~5 | NetworkBehaviour, NetworkVariable, RPC, NetworkObject patterns |
| Unity MCP tool calls (custom) | ~128 | Agentic Unity tool use: scene setup, scripting, prefabs, debugging, profiling, testing, shaders, reflection |
| Design patterns (custom) | ~75 | ScriptableObject architecture, Object Pooling, Singleton, State Machine, Observer, Strategy, Command, Factory, Flyweight, Save/Load, ServiceLocator, EventBus |
| **Total (deduped)** | **~45k** | |

### Datasets considered and rejected

| Source | Why skipped |
|---|---|
| `ibranze/codellama_unity3d_v1/v2` | 535 + 2k rows, wrong format, overlaps with vishnuOI |
| `ZorraZabb/code25pile75_shuffle_filtered_unity3d_dedup` | 1M rows of noise — substring "unity3d" filter catches Python, JS, news articles, medical papers |
| `supergoose/buzz_sources_244_unity3d-asset` | 101 rows of git diff YAML for .meta files |
| `robertzeng/unity3ddoc` | 113k rows but shallow/trivial responses (5-15 word generic definitions) |
| `DLMveloper/DLM_DataSet` | 3.6M rows but only ~3,570 "Unity"-tagged, all trivial (variable declarations, basic array ops) |
| `Saibalaji25/unity-csharp-scripts` | Requires login + T&C agreement, unknown quality, 11 downloads/mo |

## Training Configuration (M5 Pro 64GB)

```
Model:      Qwen2.5-Coder-32B-Instruct (4-bit QLoRA)
Batch size: 1
Layers:     16/64 (25% LoRA)
LR:         1e-4
Max seq:    4096
Iters:      2000
Peak mem:   ~25GB
Duration:   ~45-60 min
```

## GGUF Output Quantization

| Type | Size | Best For |
|---|---|---|
| Q4_K_M | ~20 GB | 24GB+ systems |
| Q5_K_M | ~23 GB | 32GB+ systems |
| **Q6_K** | **~27 GB** | **M5 Pro 64GB (recommended)** |
| Q8_0 | ~34 GB | 64GB+ systems (max quality) |

## Unity-MCP Bridge

Run the bridge setup to configure connections to Claude Desktop, Cursor, VS Code, or the included Python client:

```bash
python3 04_setup_unity_mcp_bridge.py
```

Then connect any MCP-compatible AI client (Claude Desktop, Cursor, VS Code with Cline/Copilot, Codex) to the Unity Editor.

## Project Structure

```
unity-agent-training/
├── 01_build_unity_mcp_dataset.py            # MCP tool-calling dataset
├── 02_build_unity_design_patterns_dataset.py  # Design patterns dataset
├── 03_train_and_export_unity_agent.py        # MLX training + GGUF export
├── 04_setup_unity_mcp_bridge.py              # Unity-MCP integration
├── 05_download_and_merge_datasets.py         # HF dataset download + merge
├── 06_extract_unity_repos.py                 # Unity org repo C# extraction
├── 07_integrate_new_datasets.py              # Newly discovered datasets
├── Modelfile                                 # Ollama model definition
├── QUICKSTART.md                             # Quick start guide
├── datasets/
│   ├── unity_mcp_tool_calls_{train,valid,test}.jsonl
│   ├── unity_design_patterns_{train,valid,test}.jsonl
│   ├── unity_api_instructions.jsonl          # (downloaded)
│   ├── unity_repos_{train,valid,test}.jsonl  # (from repo extraction)
│   ├── function_calling_full.jsonl           # (downloaded)
│   ├── train.jsonl                           # Merged MLX-ready
│   ├── valid.jsonl                           # Merged MLX-ready
│   └── test.jsonl                            # Merged MLX-ready
├── adapters/                                 # LoRA adapters (after training)
├── fused_model/                              # Fused MLX model
├── gguf/                                     # GGUF quantized models
└── mcp_config/                               # Bridge configuration
    ├── claude_desktop_config.json
    ├── vscode_mcp.json
    ├── unity_mcp_server_config.json
    ├── unity_mcp_client.py
    └── UnityAgentWindow.cs
```

```<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
make a research on internet what will be best setup for each profile i have<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Can you process this?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Research the best Hermes Agent plugin configuration for a 'game-dev' profile focused on Unity game development, 3D graphics, game logic, shaders, and game engine workflows. Consider which plugins (model providers, web search, browser, memory, tools) are most relevant for a game developer. Focus on practical recommendations: which plugins to enable, which to skip, and why. Check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.<|im_end|>


Prompt:
<|im_start|>system
You are in General Assistant profile. Use the most efficient tool for each request.
<|im_end|>
<|im_start|>user
make a research on internet what will be best setup for each profile i have<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
I need to process proc_0a497b8b5935 exited (exit code -15,. Please help.<|im_end|>


Prompt:
<|im_start|>system
You are in Data Analyst profile. Prioritize data loading, Python analysis, and charting.
<|im_end|>
<|im_start|>user
[ASYNC DELEGATION BATCH COMPLETE — deleg_2d07ecd6]
A background fan-out of 8 subagent(s) you dispatched earlier has finished. All ran in parallel and waited on each other; their consolidated results are below. You may have moved on since dispatching — act on these or re-dispatch if things have changed.

Dispatched: 2026-07-13 23:59:18 (11m48s ago)
Context you provided: Clone these 10 repos to /Volumes/this_and_that/projects/multi-repo_0/ using `git clone https://github.com/openclaw/<repo>.git`. Run from /Volumes/this_and_that/projects/multi-repo_0/. Report any failures.
Role: leaf   Model: ?   Total duration: 708.72s

--- ✓ TASK 1/8: Clone 10 OpenClaw repos: openclaw, docs, clawhub, clawgo, homebrew-tap, openclaw.ai, nix-openclaw, nix-openclaw-tools, lobster, hermit  (status=completed, api_calls=3, 356.23s) ---
Context length exceeded (379 tokens). Cannot compress further.

--- ✓ TASK 2/8: Clone 10 OpenClaw repos: clawdinators, openclaw-ansible, casa, spogo, goplaces, voice-community, acpx, community, caclawphony, discrawl  (status=completed, api_calls=4, 525.97s) ---
All 10 repos cloned successfully. Here's the summary:

**Outcome: 10/10 repos cloned successfully, 0 failures.**

All 10 OpenClaw repositories are now present at `/Volumes/this_and_that/projects/multi-repo_0/`:

| # | Repo | Status |
|---|------|--------|
| 1 | clawdinators | ✅ |
| 2 | openclaw-ansible | ✅ |
| 3 | casa | ✅ |
| 4 | spogo | ✅ |
| 5 | goplaces | ✅ |
| 6 | voice-community | ✅ |
| 7 | acpx | ✅ |
| 8 | community | ✅ |
| 9 | caclawphony | ✅ |
| 10 | discrawl | ✅ |

No issues encountered — all clones completed cleanly with exit code 0.

--- ✓ TASK 3/8: Clone 10 OpenClaw repos: slacrawl, crabline, releases, docs, esp-openclaw-node, shellbench, notcrawl, clawsweeper, clownfish, crabpot  (status=completed, api_calls=3, 394.02s) ---
All 10 repos cloned successfully. Here's the summary:

**Result: 10/10 repos cloned with zero failures.**

Cloned to `/Volumes/this_and_that/projects/multi-repo_0/`:

| # | Repo | Status |
|---|------|--------|
| 1 | `slacrawl` | ✅ Cloned |
| 2 | `crabline` | ✅ Cloned |
| 3 | `releases` | ✅ Cloned (15,075 files) |
| 4 | `docs` | ✅ Cloned |
| 5 | `esp-openclaw-node` | ✅ Cloned |
| 6 | `shellbench` | ✅ Cloned (23,628 files) |
| 7 | `notcrawl` | ✅ Cloned |
| 8 | `clawsweeper` | ✅ Cloned |
| 9 | `clownfish` | ✅ Cloned |
| 10 | `crabpot` | ✅ Cloned |

No issues encountered. All 10 repositories are present and fully cloned.

--- ✓ TASK 4/8: Clone 10 OpenClaw repos: wacrawl, plugin-inspector, kitchen-sink, gitcrawl, Kova, cookbook, clawsweeper.bot, clawbox, openclaw-rtt, crawlkit  (status=completed, api_calls=6, 621.05s) ---
Context length exceeded: max compression attempts (3) reached.

--- ✓ TASK 5/8: Clone 10 OpenClaw repos: crawlbar, clawgrit-reports, fs-safe, ask-molty, graincrawl, clickclack, clawdex, proxyline, telecrawl, clawpatch  (status=completed, api_calls=12, 708.31s) ---
All 10 repos cloned successfully. Here's the summary:

**Outcome:** All 10 OpenClaw repositories cloned successfully into `/Volumes/this_and_that/projects/multi-repo_0/`.

**Cloned repos:**
1. ✅ crawlbar
2. ✅ clawgrit-reports (previously cloned)
3. ✅ fs-safe
4. ✅ ask-molty
5. ✅ graincrawl
6. ✅ clickclack
7. ✅ clawdex
8. ✅ proxyline
9. ✅ telecrawl
10. ✅ clawpatch

**Issues:** None. All 10 clones completed with exit code 0.

--- ✓ TASK 6/8: Clone 10 OpenClaw repos: crabfleet, agent-skills, rfcs, rastermill, libopus-wasm, octopool, clawpdf, photoscrawl, clawrouter, imsgcrawl  (status=completed, api_calls=2, 318.11s) ---
All 10 repos cloned successfully. Here's the summary:

**Outcome: 10/10 repos cloned successfully, 0 failures.**

Cloned to `/Volumes/this_and_that/projects/multi-repo_0/`:

| # | Repo | Status |
|---|------|--------|
| 1 | crabfleet | ✅ |
| 2 | agent-skills | ✅ |
| 3 | rfcs | ✅ |
| 4 | rastermill | ✅ |
| 5 | libopus-wasm | ✅ |
| 6 | octopool | ✅ |
| 7 | clawpdf | ✅ |
| 8 | photoscrawl | ✅ |
| 9 | clawrouter | ✅ |
| 10 | imsgcrawl | ✅ |

No issues encountered — all clones completed cleanly with no errors.

--- ✓ TASK 7/8: Clone 10 OpenClaw repos: ffmpeg-wasm, libterminal, clawscan, uirouter, babelfish, crabhelm, turnwire, design-system, reef, AXorcist  (status=completed, api_calls=3, 326.1s) ---
Context length exceeded (435 tokens). Cannot compress further.

--- ✓ TASK 8/8: Clone remaining 8 OpenClaw repos: Peekaboo, Tachikoma, mcporter, Swabble, imsg, gogcli, wacli, songsee, butter.bot  (status=completed, api_calls=3, 355.6s) ---
Context length exceeded (1,277 tokens). Cannot compress further.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
I need to check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.. Please help.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Research the best Hermes Agent plugin configuration for an 'office-manager' profile focused on office productivity, document management, spreadsheets, presentations, email management, scheduling, and administrative tasks. Consider which plugins (model providers, web search, browser, memory, tools) are most relevant. Focus on practical recommendations: which plugins to enable, which to skip, and why. Check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.<|im_end|>


Prompt:
<|im_start|>system
You are in Creative profile. Prioritize content generation and design-oriented tools.
<|im_end|>
<|im_start|>user
my laptop crashed. Continue where we left or in case something went corrupt then lets start over<|im_end|>


Prompt:
<|im_start|>system
You are in Creative profile. Prioritize content generation and design-oriented tools.
<|im_end|>
<|im_start|>user
progress?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
I need to process this. Please help.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
lower iter so we won't hit OOM<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Research the best Hermes Agent plugin configuration for a 'game-dev' profile focused on Unity game development, 3D graphics, game logic, shaders, and game engine workflows. Consider which plugins (model providers, web search, browser, memory, tools) are most relevant for a game developer. Focus on practical recommendations: which plugins to enable, which to skip, and why. Check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
I need to process this. Please help.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
i need to know if you are making a GGUF model or MLX model?<|im_end|>


Prompt:
<|im_start|>system
You are in Developer profile. Prioritize code execution, file editing, and terminal access.
<|im_end|>
<|im_start|>user
Research the best Hermes Agent plugin configuration for an 'image-generator' profile focused on AI image generation, image editing, design workflows, and visual content creation. Consider which plugins (model providers, web search, browser, memory, tools) are most relevant for an image generation agent. Focus on practical recommendations: which plugins to enable, which to skip, and why. Check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Can you process this?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Research the best Hermes Agent plugin configuration for an 'office-manager' profile focused on office productivity, document management, spreadsheets, presentations, email management, scheduling, and administrative tasks. Consider which plugins (model providers, web search, browser, memory, tools) are most relevant. Focus on practical recommendations: which plugins to enable, which to skip, and why. Check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
i need to know if you are making a GGUF model or MLX model?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
lower iter so we won't hit OOM<|im_end|>


Prompt:
<|im_start|>system
You are in System Admin profile. Prioritize terminal commands, process management, and system tools.
<|im_end|>
<|im_start|>user
progress?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
[IMPORTANT: Background process proc_0a497b8b5935 exited (exit code -15, SIGTERM).
Command: unset PYTHONPATH; /Users/jacobeen/.hermes/hermes-agent/venv/bin/python3 03_train_and_export_unity_agent.py train 2>&1
Output:
4<00:26,  1.25s/it]
Calculating loss...:  20%|██        | 5/25 [00:04<00:21,  1.08s/it]
Calculating loss...:  24%|██▍       | 6/25 [00:05<00:16,  1.18it/s]
Calculating loss...:  28%|██▊       | 7/25 [00:06<00:16,  1.07it/s]
Calculating loss...:  32%|███▏      | 8/25 [00:07<00:15,  1.07it/s]
Calculating loss...:  36%|███▌      | 9/25 [00:08<00:17,  1.10s/it]
Calculating loss...:  40%|████      | 10/25 [00:10<00:19,  1.27s/it]
Calculating loss...:  44%|████▍     | 11/25 [00:11<00:17,  1.28s/it]
Calculating loss...:  48%|████▊     | 12/25 [00:12<00:14,  1.13s/it]
Calculating loss...:  52%|█████▏    | 13/25 [00:14<00:14,  1.24s/it]
Calculating loss...:  56%|█████▌    | 14/25 [00:14<00:10,  1.02it/s]
Calculating loss...:  60%|██████    | 15/25 [00:14<00:08,  1.24it/s]
Calculating loss...:  64%|██████▍   | 16/25 [00:16<00:09,  1.00s/it]
Calculating loss...:  68%|██████▊   | 17/25 [00:16<00:07,  1.13it/s]
Calculating loss...:  72%|███████▏  | 18/25 [00:17<00:06,  1.11it/s]
Calculating loss...:  76%|███████▌  | 19/25 [00:18<00:04,  1.25it/s]
Calculating loss...:  80%|████████  | 20/25 [00:18<00:03,  1.46it/s]
Calculating loss...:  84%|████████▍ | 21/25 [00:19<00:03,  1.33it/s]
Calculating loss...:  88%|████████▊ | 22/25 [00:20<00:02,  1.24it/s]
Calculating loss...:  92%|█████████▏| 23/25 [00:21<00:01,  1.26it/s]
Calculating loss...:  96%|█████████▌| 24/25 [00:22<00:00,  1.27it/s]
Calculating loss...: 100%|██████████| 25/25 [00:24<00:00,  1.10s/it]
Calculating loss...: 100%|██████████| 25/25 [00:24<00:00,  1.04it/s]
Iter 100: Val loss nan, Val took 24.072s
Iter 100: Train loss nan, Learning Rate 1.000e-04, It/sec 0.478, Tokens/sec 10225505.663, Trained Tokens 1068717925, Peak mem 65.672 GB
/Users/jacobeen/.local/share/uv/python/cpython-3.11.15-macos-aarch64-none/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
]<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
[ASYNC DELEGATION BATCH COMPLETE — deleg_2d07ecd6]
A background fan-out of 8 subagent(s) you dispatched earlier has finished. All ran in parallel and waited on each other; their consolidated results are below. You may have moved on since dispatching — act on these or re-dispatch if things have changed.

Dispatched: 2026-07-13 23:59:18 (11m48s ago)
Context you provided: Clone these 10 repos to /Volumes/this_and_that/projects/multi-repo_0/ using `git clone https://github.com/openclaw/<repo>.git`. Run from /Volumes/this_and_that/projects/multi-repo_0/. Report any failures.
Role: leaf   Model: ?   Total duration: 708.72s

--- ✓ TASK 1/8: Clone 10 OpenClaw repos: openclaw, docs, clawhub, clawgo, homebrew-tap, openclaw.ai, nix-openclaw, nix-openclaw-tools, lobster, hermit  (status=completed, api_calls=3, 356.23s) ---
Context length exceeded (379 tokens). Cannot compress further.

--- ✓ TASK 2/8: Clone 10 OpenClaw repos: clawdinators, openclaw-ansible, casa, spogo, goplaces, voice-community, acpx, community, caclawphony, discrawl  (status=completed, api_calls=4, 525.97s) ---
All 10 repos cloned successfully. Here's the summary:

**Outcome: 10/10 repos cloned successfully, 0 failures.**

All 10 OpenClaw repositories are now present at `/Volumes/this_and_that/projects/multi-repo_0/`:

| # | Repo | Status |
|---|------|--------|
| 1 | clawdinators | ✅ |
| 2 | openclaw-ansible | ✅ |
| 3 | casa | ✅ |
| 4 | spogo | ✅ |
| 5 | goplaces | ✅ |
| 6 | voice-community | ✅ |
| 7 | acpx | ✅ |
| 8 | community | ✅ |
| 9 | caclawphony | ✅ |
| 10 | discrawl | ✅ |

No issues encountered — all clones completed cleanly with exit code 0.

--- ✓ TASK 3/8: Clone 10 OpenClaw repos: slacrawl, crabline, releases, docs, esp-openclaw-node, shellbench, notcrawl, clawsweeper, clownfish, crabpot  (status=completed, api_calls=3, 394.02s) ---
All 10 repos cloned successfully. Here's the summary:

**Result: 10/10 repos cloned with zero failures.**

Cloned to `/Volumes/this_and_that/projects/multi-repo_0/`:

| # | Repo | Status |
|---|------|--------|
| 1 | `slacrawl` | ✅ Cloned |
| 2 | `crabline` | ✅ Cloned |
| 3 | `releases` | ✅ Cloned (15,075 files) |
| 4 | `docs` | ✅ Cloned |
| 5 | `esp-openclaw-node` | ✅ Cloned |
| 6 | `shellbench` | ✅ Cloned (23,628 files) |
| 7 | `notcrawl` | ✅ Cloned |
| 8 | `clawsweeper` | ✅ Cloned |
| 9 | `clownfish` | ✅ Cloned |
| 10 | `crabpot` | ✅ Cloned |

No issues encountered. All 10 repositories are present and fully cloned.

--- ✓ TASK 4/8: Clone 10 OpenClaw repos: wacrawl, plugin-inspector, kitchen-sink, gitcrawl, Kova, cookbook, clawsweeper.bot, clawbox, openclaw-rtt, crawlkit  (status=completed, api_calls=6, 621.05s) ---
Context length exceeded: max compression attempts (3) reached.

--- ✓ TASK 5/8: Clone 10 OpenClaw repos: crawlbar, clawgrit-reports, fs-safe, ask-molty, graincrawl, clickclack, clawdex, proxyline, telecrawl, clawpatch  (status=completed, api_calls=12, 708.31s) ---
All 10 repos cloned successfully. Here's the summary:

**Outcome:** All 10 OpenClaw repositories cloned successfully into `/Volumes/this_and_that/projects/multi-repo_0/`.

**Cloned repos:**
1. ✅ crawlbar
2. ✅ clawgrit-reports (previously cloned)
3. ✅ fs-safe
4. ✅ ask-molty
5. ✅ graincrawl
6. ✅ clickclack
7. ✅ clawdex
8. ✅ proxyline
9. ✅ telecrawl
10. ✅ clawpatch

**Issues:** None. All 10 clones completed with exit code 0.

--- ✓ TASK 6/8: Clone 10 OpenClaw repos: crabfleet, agent-skills, rfcs, rastermill, libopus-wasm, octopool, clawpdf, photoscrawl, clawrouter, imsgcrawl  (status=completed, api_calls=2, 318.11s) ---
All 10 repos cloned successfully. Here's the summary:

**Outcome: 10/10 repos cloned successfully, 0 failures.**

Cloned to `/Volumes/this_and_that/projects/multi-repo_0/`:

| # | Repo | Status |
|---|------|--------|
| 1 | crabfleet | ✅ |
| 2 | agent-skills | ✅ |
| 3 | rfcs | ✅ |
| 4 | rastermill | ✅ |
| 5 | libopus-wasm | ✅ |
| 6 | octopool | ✅ |
| 7 | clawpdf | ✅ |
| 8 | photoscrawl | ✅ |
| 9 | clawrouter | ✅ |
| 10 | imsgcrawl | ✅ |

No issues encountered — all clones completed cleanly with no errors.

--- ✓ TASK 7/8: Clone 10 OpenClaw repos: ffmpeg-wasm, libterminal, clawscan, uirouter, babelfish, crabhelm, turnwire, design-system, reef, AXorcist  (status=completed, api_calls=3, 326.1s) ---
Context length exceeded (435 tokens). Cannot compress further.

--- ✓ TASK 8/8: Clone remaining 8 OpenClaw repos: Peekaboo, Tachikoma, mcporter, Swabble, imsg, gogcli, wacli, songsee, butter.bot  (status=completed, api_calls=3, 355.6s) ---
Context length exceeded (1,277 tokens). Cannot compress further.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
[IMPORTANT: Background process proc_0a497b8b5935 exited (exit code -15, SIGTERM).
Command: unset PYTHONPATH; /Users/jacobeen/.hermes/hermes-agent/venv/bin/python3 03_train_and_export_unity_agent.py train 2>&1
Output:
4<00:26,  1.25s/it]
Calculating loss...:  20%|██        | 5/25 [00:04<00:21,  1.08s/it]
Calculating loss...:  24%|██▍       | 6/25 [00:05<00:16,  1.18it/s]
Calculating loss...:  28%|██▊       | 7/25 [00:06<00:16,  1.07it/s]
Calculating loss...:  32%|███▏      | 8/25 [00:07<00:15,  1.07it/s]
Calculating loss...:  36%|███▌      | 9/25 [00:08<00:17,  1.10s/it]
Calculating loss...:  40%|████      | 10/25 [00:10<00:19,  1.27s/it]
Calculating loss...:  44%|████▍     | 11/25 [00:11<00:17,  1.28s/it]
Calculating loss...:  48%|████▊     | 12/25 [00:12<00:14,  1.13s/it]
Calculating loss...:  52%|█████▏    | 13/25 [00:14<00:14,  1.24s/it]
Calculating loss...:  56%|█████▌    | 14/25 [00:14<00:10,  1.02it/s]
Calculating loss...:  60%|██████    | 15/25 [00:14<00:08,  1.24it/s]
Calculating loss...:  64%|██████▍   | 16/25 [00:16<00:09,  1.00s/it]
Calculating loss...:  68%|██████▊   | 17/25 [00:16<00:07,  1.13it/s]
Calculating loss...:  72%|███████▏  | 18/25 [00:17<00:06,  1.11it/s]
Calculating loss...:  76%|███████▌  | 19/25 [00:18<00:04,  1.25it/s]
Calculating loss...:  80%|████████  | 20/25 [00:18<00:03,  1.46it/s]
Calculating loss...:  84%|████████▍ | 21/25 [00:19<00:03,  1.33it/s]
Calculating loss...:  88%|████████▊ | 22/25 [00:20<00:02,  1.24it/s]
Calculating loss...:  92%|█████████▏| 23/25 [00:21<00:01,  1.26it/s]
Calculating loss...:  96%|█████████▌| 24/25 [00:22<00:00,  1.27it/s]
Calculating loss...: 100%|██████████| 25/25 [00:24<00:00,  1.10s/it]
Calculating loss...: 100%|██████████| 25/25 [00:24<00:00,  1.04it/s]
Iter 100: Val loss nan, Val took 24.072s
Iter 100: Train loss nan, Learning Rate 1.000e-04, It/sec 0.478, Tokens/sec 10225505.663, Trained Tokens 1068717925, Peak mem 65.672 GB
/Users/jacobeen/.local/share/uv/python/cpython-3.11.15-macos-aarch64-none/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
]<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
i need to know if you are making a GGUF model or MLX model?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Can you process this?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
i need to know if you are making a GGUF model or MLX model?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
which custom made model on HuggingFace is extraordinarly smart?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
I need to process this. Please help.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
execute this project plan

--- Attached Context ---

📄 @file:`QUICKSTART.md` (551 tokens)
```markdown
# Quick Start

## 1. Run the Fine-Tuned Model

### MLX (recommended for inference):
```bash
mlx_lm.generate --model ./fused_model --prompt "Your Unity task..."
```

### Ollama:
Create Modelfile in this directory:
```dockerfile
FROM ./gguf/unity-agent-Q6_K.gguf
TEMPLATE "{{ .Prompt }}"
PARAMETER temperature 0.3
PARAMETER num_ctx 4096
```

Then:
```bash
ollama create unity-agent -f Modelfile
ollama serve
```

### LM Studio:
- Open LM Studio → "My Models" → "Add Model"
- Select ./gguf/unity-agent-Q6_K.gguf
- Start server on port 1234

## 2. Connect to Unity Editor

### Option A: Unity-MCP CLI (recommended)
```bash
npx -y @aigamedeveloper/mcp-server
# Connects Claude Desktop, Cursor, or VS Code to Unity Editor
```

### Option B: Custom Editor Window
- Copy `mcp_config/UnityAgentWindow.cs` to your project's `Assets/Editor/`
- Window → Unity Agent → Chat

### Option C: Python client
```bash
python3 mcp_config/unity_mcp_client.py --list
python3 mcp_config/unity_mcp_client.py --tool gameobject-create \
  --args '{"name":"Cube","primitiveType":"Cube"}'
```

## 3. Example Workflows

**Create a scene from scratch:**
> "Create a medieval village with 5 houses, a central fountain, and a player character"

**Fix bugs interactively:**
> "Read the console logs and fix the NullReferenceException in PlayerHealth.cs"

**Profile and optimize:**
> "Profile the game, find the bottleneck, and suggest optimizations"

**Design a system:**
> "Create a ScriptableObject-based inventory system with item pickup, UI display, and save/load"

## 4. Dataset Pipeline (full run)

```bash
# Generate custom datasets
python3 01_build_unity_mcp_dataset.py
python3 02_build_unity_design_patterns_dataset.py

# Download external datasets
python3 05_download_and_merge_datasets.py
python3 06_extract_unity_repos.py
python3 07_integrate_new_datasets.py

# Train + export
python3 03_train_and_export_unity_agent.py
```

## 5. Fine-Tuning Stages

```bash
# Prepare only (merge datasets)
python3 03_train_and_export_unity_agent.py --stage prepare

# Train only (QLoRA)
python3 03_train_and_export_unity_agent.py --stage train

# Fuse adapters and export GGUF
python3 03_train_and_export_unity_agent.py --stage export
```

```

📄 @file:`README.md` (1740 tokens)
```markdown
# Unity Agent Developer — Model Training Pipeline

Build and deploy a Unity game development agent model on Apple Silicon (M5 Pro).

## Quick Start

```bash
# 1. Generate custom datasets
python3 01_build_unity_mcp_dataset.py
python3 02_build_unity_design_patterns_dataset.py

# 2. Download + merge everything (internet needed)
python3 05_download_and_merge_datasets.py       # vishnuOI + function-calling
python3 06_extract_unity_repos.py                # ML-Agents, ECS, uGUI, netcode
python3 07_integrate_new_datasets.py             # Erocal Q&A + Mix & Jam mechanics

# 3. Fine-tune (MLX QLoRA on Apple Silicon)
python3 03_train_and_export_unity_agent.py

# 4. Export to GGUF
python3 03_train_and_export_unity_agent.py --stage export

# 5. Run
ollama create unity-agent -f Modelfile
ollama run unity-agent
```

## Components

| # | Script | What It Does | Output |
|---|---|---|---|
| 01 | `01_build_unity_mcp_dataset.py` | Generates ~128 MCP tool-calling examples from real game-dev tasks | `datasets/unity_mcp_tool_calls_*.jsonl` |
| 02 | `02_build_unity_design_patterns_dataset.py` | Generates ~75 design pattern + bug-fix examples | `datasets/unity_design_patterns_*.jsonl` |
| 03 | `03_train_and_export_unity_agent.py` | MLX QLoRA train → fuse → GGUF export → eval | `adapters/`, `fused_model/`, `gguf/` |
| 04 | `04_setup_unity_mcp_bridge.py` | Sets up Unity-MCP bridge, configs for Claude/Cursor/VS Code | `mcp_config/` |
| 05 | `05_download_and_merge_datasets.py` | Downloads `vishnuOI/unity-dev-instructions` (46k) + `hermes-function-calling-v1` (2k), merges everything | `datasets/train.jsonl` |
| 06 | `06_extract_unity_repos.py` | Extracts C# code from Unity repos: ML-Agents (29), ECS (10), uGUI (10), Netcode (5) | `datasets/unity_repos_*.jsonl` |
| 07 | `07_integrate_new_datasets.py` | Downloads + integrates `Erocal/Unity_ResolvedQuestions` (400 real Q&A) and `MixAndJam/GameMechanics` (32 game mechanic C# codes) | Rebuilds `datasets/{train,valid,test}.jsonl` |

## Dataset Composition (after all merges)

| Source | Rows | Coverage |
|---|---|---|
| `vishnuOI/unity-dev-instructions` | ~42k (after dedup) | Unity API docs, Stack Overflow, GitHub code — scripting, rendering, physics, editor, XR, UI, animation, networking, math, performance, input |
| `NousResearch/hermes-function-calling-v1` | ~1.8k | Generic tool-calling / JSON-mode structured outputs |
| `Erocal/Unity_ResolvedQuestions` | ~400 | Real resolved Q&A — physics, ECS, URP/HDRP, UI Toolkit, editor tooling, platform issues, animation rigging |
| `AmnaHassan/MixAndJam-GameMechanics` | ~32 | Full game mechanic C# implementations: Celeste movement, Splatoon ink, Arkham combat, God of War axe throw, Balatro feel, AC dialogue, Luigi's Mansion ghost capture, Mario Galaxy launch stars, Pikmin gameplay |
| ML-Agents example agents | ~29 | RL agent patterns (Crawler, Walker, FoodCollector, Soccer, Pyramids, WallJump, Worm, Hallway, 3DBall, PushBlock, GridWorld, Sorter) |
| ML-Agents core + sensors | ~5 | Agent.cs lifecycle, ray perception, camera sensor, vector sensor, grid sensor |
| ECS/DOTS samples | ~10 | Jobs, SystemBase, ISystem, IComponentData, IAspect, Baker patterns |
| uGUI | ~10 | Graphic, Image, Text, Button, Slider, Toggle, ScrollRect, Dropdown, InputField, LayoutGroup |
| Netcode for GameObjects | ~5 | NetworkBehaviour, NetworkVariable, RPC, NetworkObject patterns |
| Unity MCP tool calls (custom) | ~128 | Agentic Unity tool use: scene setup, scripting, prefabs, debugging, profiling, testing, shaders, reflection |
| Design patterns (custom) | ~75 | ScriptableObject architecture, Object Pooling, Singleton, State Machine, Observer, Strategy, Command, Factory, Flyweight, Save/Load, ServiceLocator, EventBus |
| **Total (deduped)** | **~45k** | |

### Datasets considered and rejected

| Source | Why skipped |
|---|---|
| `ibranze/codellama_unity3d_v1/v2` | 535 + 2k rows, wrong format, overlaps with vishnuOI |
| `ZorraZabb/code25pile75_shuffle_filtered_unity3d_dedup` | 1M rows of noise — substring "unity3d" filter catches Python, JS, news articles, medical papers |
| `supergoose/buzz_sources_244_unity3d-asset` | 101 rows of git diff YAML for .meta files |
| `robertzeng/unity3ddoc` | 113k rows but shallow/trivial responses (5-15 word generic definitions) |
| `DLMveloper/DLM_DataSet` | 3.6M rows but only ~3,570 "Unity"-tagged, all trivial (variable declarations, basic array ops) |
| `Saibalaji25/unity-csharp-scripts` | Requires login + T&C agreement, unknown quality, 11 downloads/mo |

## Training Configuration (M5 Pro 64GB)

```
Model:      Qwen2.5-Coder-32B-Instruct (4-bit QLoRA)
Batch size: 1
Layers:     16/64 (25% LoRA)
LR:         1e-4
Max seq:    4096
Iters:      2000
Peak mem:   ~25GB
Duration:   ~45-60 min
```

## GGUF Output Quantization

| Type | Size | Best For |
|---|---|---|
| Q4_K_M | ~20 GB | 24GB+ systems |
| Q5_K_M | ~23 GB | 32GB+ systems |
| **Q6_K** | **~27 GB** | **M5 Pro 64GB (recommended)** |
| Q8_0 | ~34 GB | 64GB+ systems (max quality) |

## Unity-MCP Bridge

Run the bridge setup to configure connections to Claude Desktop, Cursor, VS Code, or the included Python client:

```bash
python3 04_setup_unity_mcp_bridge.py
```

Then connect any MCP-compatible AI client (Claude Desktop, Cursor, VS Code with Cline/Copilot, Codex) to the Unity Editor.

## Project Structure

```
unity-agent-training/
├── 01_build_unity_mcp_dataset.py            # MCP tool-calling dataset
├── 02_build_unity_design_patterns_dataset.py  # Design patterns dataset
├── 03_train_and_export_unity_agent.py        # MLX training + GGUF export
├── 04_setup_unity_mcp_bridge.py              # Unity-MCP integration
├── 05_download_and_merge_datasets.py         # HF dataset download + merge
├── 06_extract_unity_repos.py                 # Unity org repo C# extraction
├── 07_integrate_new_datasets.py              # Newly discovered datasets
├── Modelfile                                 # Ollama model definition
├── QUICKSTART.md                             # Quick start guide
├── datasets/
│   ├── unity_mcp_tool_calls_{train,valid,test}.jsonl
│   ├── unity_design_patterns_{train,valid,test}.jsonl
│   ├── unity_api_instructions.jsonl          # (downloaded)
│   ├── unity_repos_{train,valid,test}.jsonl  # (from repo extraction)
│   ├── function_calling_full.jsonl           # (downloaded)
│   ├── train.jsonl                           # Merged MLX-ready
│   ├── valid.jsonl                           # Merged MLX-ready
│   └── test.jsonl                            # Merged MLX-ready
├── adapters/                                 # LoRA adapters (after training)
├── fused_model/                              # Fused MLX model
├── gguf/                                     # GGUF quantized models
└── mcp_config/                               # Bridge configuration
    ├── claude_desktop_config.json
    ├── vscode_mcp.json
    ├── unity_mcp_server_config.json
    ├── unity_mcp_client.py
    └── UnityAgentWindow.cs
```

```<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
I need to check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.. Please help.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
which custom made model on HuggingFace is extraordinarly smart?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
[IMPORTANT: Background process proc_0a497b8b5935 exited (exit code -15, SIGTERM).
Command: unset PYTHONPATH; /Users/jacobeen/.hermes/hermes-agent/venv/bin/python3 03_train_and_export_unity_agent.py train 2>&1
Output:
4<00:26,  1.25s/it]
Calculating loss...:  20%|██        | 5/25 [00:04<00:21,  1.08s/it]
Calculating loss...:  24%|██▍       | 6/25 [00:05<00:16,  1.18it/s]
Calculating loss...:  28%|██▊       | 7/25 [00:06<00:16,  1.07it/s]
Calculating loss...:  32%|███▏      | 8/25 [00:07<00:15,  1.07it/s]
Calculating loss...:  36%|███▌      | 9/25 [00:08<00:17,  1.10s/it]
Calculating loss...:  40%|████      | 10/25 [00:10<00:19,  1.27s/it]
Calculating loss...:  44%|████▍     | 11/25 [00:11<00:17,  1.28s/it]
Calculating loss...:  48%|████▊     | 12/25 [00:12<00:14,  1.13s/it]
Calculating loss...:  52%|█████▏    | 13/25 [00:14<00:14,  1.24s/it]
Calculating loss...:  56%|█████▌    | 14/25 [00:14<00:10,  1.02it/s]
Calculating loss...:  60%|██████    | 15/25 [00:14<00:08,  1.24it/s]
Calculating loss...:  64%|██████▍   | 16/25 [00:16<00:09,  1.00s/it]
Calculating loss...:  68%|██████▊   | 17/25 [00:16<00:07,  1.13it/s]
Calculating loss...:  72%|███████▏  | 18/25 [00:17<00:06,  1.11it/s]
Calculating loss...:  76%|███████▌  | 19/25 [00:18<00:04,  1.25it/s]
Calculating loss...:  80%|████████  | 20/25 [00:18<00:03,  1.46it/s]
Calculating loss...:  84%|████████▍ | 21/25 [00:19<00:03,  1.33it/s]
Calculating loss...:  88%|████████▊ | 22/25 [00:20<00:02,  1.24it/s]
Calculating loss...:  92%|█████████▏| 23/25 [00:21<00:01,  1.26it/s]
Calculating loss...:  96%|█████████▌| 24/25 [00:22<00:00,  1.27it/s]
Calculating loss...: 100%|██████████| 25/25 [00:24<00:00,  1.10s/it]
Calculating loss...: 100%|██████████| 25/25 [00:24<00:00,  1.04it/s]
Iter 100: Val loss nan, Val took 24.072s
Iter 100: Train loss nan, Learning Rate 1.000e-04, It/sec 0.478, Tokens/sec 10225505.663, Trained Tokens 1068717925, Peak mem 65.672 GB
/Users/jacobeen/.local/share/uv/python/cpython-3.11.15-macos-aarch64-none/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
]<|im_end|>


Prompt:
<|im_start|>system
You are in Developer profile. Prioritize code execution, file editing, and terminal access.
<|im_end|>
<|im_start|>user
execute this project plan

--- Attached Context ---

📄 @file:`QUICKSTART.md` (551 tokens)
```markdown
# Quick Start

## 1. Run the Fine-Tuned Model

### MLX (recommended for inference):
```bash
mlx_lm.generate --model ./fused_model --prompt "Your Unity task..."
```

### Ollama:
Create Modelfile in this directory:
```dockerfile
FROM ./gguf/unity-agent-Q6_K.gguf
TEMPLATE "{{ .Prompt }}"
PARAMETER temperature 0.3
PARAMETER num_ctx 4096
```

Then:
```bash
ollama create unity-agent -f Modelfile
ollama serve
```

### LM Studio:
- Open LM Studio → "My Models" → "Add Model"
- Select ./gguf/unity-agent-Q6_K.gguf
- Start server on port 1234

## 2. Connect to Unity Editor

### Option A: Unity-MCP CLI (recommended)
```bash
npx -y @aigamedeveloper/mcp-server
# Connects Claude Desktop, Cursor, or VS Code to Unity Editor
```

### Option B: Custom Editor Window
- Copy `mcp_config/UnityAgentWindow.cs` to your project's `Assets/Editor/`
- Window → Unity Agent → Chat

### Option C: Python client
```bash
python3 mcp_config/unity_mcp_client.py --list
python3 mcp_config/unity_mcp_client.py --tool gameobject-create \
  --args '{"name":"Cube","primitiveType":"Cube"}'
```

## 3. Example Workflows

**Create a scene from scratch:**
> "Create a medieval village with 5 houses, a central fountain, and a player character"

**Fix bugs interactively:**
> "Read the console logs and fix the NullReferenceException in PlayerHealth.cs"

**Profile and optimize:**
> "Profile the game, find the bottleneck, and suggest optimizations"

**Design a system:**
> "Create a ScriptableObject-based inventory system with item pickup, UI display, and save/load"

## 4. Dataset Pipeline (full run)

```bash
# Generate custom datasets
python3 01_build_unity_mcp_dataset.py
python3 02_build_unity_design_patterns_dataset.py

# Download external datasets
python3 05_download_and_merge_datasets.py
python3 06_extract_unity_repos.py
python3 07_integrate_new_datasets.py

# Train + export
python3 03_train_and_export_unity_agent.py
```

## 5. Fine-Tuning Stages

```bash
# Prepare only (merge datasets)
python3 03_train_and_export_unity_agent.py --stage prepare

# Train only (QLoRA)
python3 03_train_and_export_unity_agent.py --stage train

# Fuse adapters and export GGUF
python3 03_train_and_export_unity_agent.py --stage export
```

```

📄 @file:`README.md` (1740 tokens)
```markdown
# Unity Agent Developer — Model Training Pipeline

Build and deploy a Unity game development agent model on Apple Silicon (M5 Pro).

## Quick Start

```bash
# 1. Generate custom datasets
python3 01_build_unity_mcp_dataset.py
python3 02_build_unity_design_patterns_dataset.py

# 2. Download + merge everything (internet needed)
python3 05_download_and_merge_datasets.py       # vishnuOI + function-calling
python3 06_extract_unity_repos.py                # ML-Agents, ECS, uGUI, netcode
python3 07_integrate_new_datasets.py             # Erocal Q&A + Mix & Jam mechanics

# 3. Fine-tune (MLX QLoRA on Apple Silicon)
python3 03_train_and_export_unity_agent.py

# 4. Export to GGUF
python3 03_train_and_export_unity_agent.py --stage export

# 5. Run
ollama create unity-agent -f Modelfile
ollama run unity-agent
```

## Components

| # | Script | What It Does | Output |
|---|---|---|---|
| 01 | `01_build_unity_mcp_dataset.py` | Generates ~128 MCP tool-calling examples from real game-dev tasks | `datasets/unity_mcp_tool_calls_*.jsonl` |
| 02 | `02_build_unity_design_patterns_dataset.py` | Generates ~75 design pattern + bug-fix examples | `datasets/unity_design_patterns_*.jsonl` |
| 03 | `03_train_and_export_unity_agent.py` | MLX QLoRA train → fuse → GGUF export → eval | `adapters/`, `fused_model/`, `gguf/` |
| 04 | `04_setup_unity_mcp_bridge.py` | Sets up Unity-MCP bridge, configs for Claude/Cursor/VS Code | `mcp_config/` |
| 05 | `05_download_and_merge_datasets.py` | Downloads `vishnuOI/unity-dev-instructions` (46k) + `hermes-function-calling-v1` (2k), merges everything | `datasets/train.jsonl` |
| 06 | `06_extract_unity_repos.py` | Extracts C# code from Unity repos: ML-Agents (29), ECS (10), uGUI (10), Netcode (5) | `datasets/unity_repos_*.jsonl` |
| 07 | `07_integrate_new_datasets.py` | Downloads + integrates `Erocal/Unity_ResolvedQuestions` (400 real Q&A) and `MixAndJam/GameMechanics` (32 game mechanic C# codes) | Rebuilds `datasets/{train,valid,test}.jsonl` |

## Dataset Composition (after all merges)

| Source | Rows | Coverage |
|---|---|---|
| `vishnuOI/unity-dev-instructions` | ~42k (after dedup) | Unity API docs, Stack Overflow, GitHub code — scripting, rendering, physics, editor, XR, UI, animation, networking, math, performance, input |
| `NousResearch/hermes-function-calling-v1` | ~1.8k | Generic tool-calling / JSON-mode structured outputs |
| `Erocal/Unity_ResolvedQuestions` | ~400 | Real resolved Q&A — physics, ECS, URP/HDRP, UI Toolkit, editor tooling, platform issues, animation rigging |
| `AmnaHassan/MixAndJam-GameMechanics` | ~32 | Full game mechanic C# implementations: Celeste movement, Splatoon ink, Arkham combat, God of War axe throw, Balatro feel, AC dialogue, Luigi's Mansion ghost capture, Mario Galaxy launch stars, Pikmin gameplay |
| ML-Agents example agents | ~29 | RL agent patterns (Crawler, Walker, FoodCollector, Soccer, Pyramids, WallJump, Worm, Hallway, 3DBall, PushBlock, GridWorld, Sorter) |
| ML-Agents core + sensors | ~5 | Agent.cs lifecycle, ray perception, camera sensor, vector sensor, grid sensor |
| ECS/DOTS samples | ~10 | Jobs, SystemBase, ISystem, IComponentData, IAspect, Baker patterns |
| uGUI | ~10 | Graphic, Image, Text, Button, Slider, Toggle, ScrollRect, Dropdown, InputField, LayoutGroup |
| Netcode for GameObjects | ~5 | NetworkBehaviour, NetworkVariable, RPC, NetworkObject patterns |
| Unity MCP tool calls (custom) | ~128 | Agentic Unity tool use: scene setup, scripting, prefabs, debugging, profiling, testing, shaders, reflection |
| Design patterns (custom) | ~75 | ScriptableObject architecture, Object Pooling, Singleton, State Machine, Observer, Strategy, Command, Factory, Flyweight, Save/Load, ServiceLocator, EventBus |
| **Total (deduped)** | **~45k** | |

### Datasets considered and rejected

| Source | Why skipped |
|---|---|
| `ibranze/codellama_unity3d_v1/v2` | 535 + 2k rows, wrong format, overlaps with vishnuOI |
| `ZorraZabb/code25pile75_shuffle_filtered_unity3d_dedup` | 1M rows of noise — substring "unity3d" filter catches Python, JS, news articles, medical papers |
| `supergoose/buzz_sources_244_unity3d-asset` | 101 rows of git diff YAML for .meta files |
| `robertzeng/unity3ddoc` | 113k rows but shallow/trivial responses (5-15 word generic definitions) |
| `DLMveloper/DLM_DataSet` | 3.6M rows but only ~3,570 "Unity"-tagged, all trivial (variable declarations, basic array ops) |
| `Saibalaji25/unity-csharp-scripts` | Requires login + T&C agreement, unknown quality, 11 downloads/mo |

## Training Configuration (M5 Pro 64GB)

```
Model:      Qwen2.5-Coder-32B-Instruct (4-bit QLoRA)
Batch size: 1
Layers:     16/64 (25% LoRA)
LR:         1e-4
Max seq:    4096
Iters:      2000
Peak mem:   ~25GB
Duration:   ~45-60 min
```

## GGUF Output Quantization

| Type | Size | Best For |
|---|---|---|
| Q4_K_M | ~20 GB | 24GB+ systems |
| Q5_K_M | ~23 GB | 32GB+ systems |
| **Q6_K** | **~27 GB** | **M5 Pro 64GB (recommended)** |
| Q8_0 | ~34 GB | 64GB+ systems (max quality) |

## Unity-MCP Bridge

Run the bridge setup to configure connections to Claude Desktop, Cursor, VS Code, or the included Python client:

```bash
python3 04_setup_unity_mcp_bridge.py
```

Then connect any MCP-compatible AI client (Claude Desktop, Cursor, VS Code with Cline/Copilot, Codex) to the Unity Editor.

## Project Structure

```
unity-agent-training/
├── 01_build_unity_mcp_dataset.py            # MCP tool-calling dataset
├── 02_build_unity_design_patterns_dataset.py  # Design patterns dataset
├── 03_train_and_export_unity_agent.py        # MLX training + GGUF export
├── 04_setup_unity_mcp_bridge.py              # Unity-MCP integration
├── 05_download_and_merge_datasets.py         # HF dataset download + merge
├── 06_extract_unity_repos.py                 # Unity org repo C# extraction
├── 07_integrate_new_datasets.py              # Newly discovered datasets
├── Modelfile                                 # Ollama model definition
├── QUICKSTART.md                             # Quick start guide
├── datasets/
│   ├── unity_mcp_tool_calls_{train,valid,test}.jsonl
│   ├── unity_design_patterns_{train,valid,test}.jsonl
│   ├── unity_api_instructions.jsonl          # (downloaded)
│   ├── unity_repos_{train,valid,test}.jsonl  # (from repo extraction)
│   ├── function_calling_full.jsonl           # (downloaded)
│   ├── train.jsonl                           # Merged MLX-ready
│   ├── valid.jsonl                           # Merged MLX-ready
│   └── test.jsonl                            # Merged MLX-ready
├── adapters/                                 # LoRA adapters (after training)
├── fused_model/                              # Fused MLX model
├── gguf/                                     # GGUF quantized models
└── mcp_config/                               # Bridge configuration
    ├── claude_desktop_config.json
    ├── vscode_mcp.json
    ├── unity_mcp_server_config.json
    ├── unity_mcp_client.py
    └── UnityAgentWindow.cs
```

```<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Can you process this?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
can you make sure that every single profile is utilizing our Obsidian.md, Hindsight and othr plugins?

[System: The active model for this chat has changed to Ornith-1.0-35B-MTPLX via provider omlx-mbp. From this point forward, use this runtime metadata when answering questions about what model/provider is active.]

can you make sure that every single profile is utilizing our Obsidian.md, Hindsight and othr plugins?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Research the best Hermes Agent plugin configuration for an 'image-generator' profile focused on AI image generation, image editing, design workflows, and visual content creation. Consider which plugins (model providers, web search, browser, memory, tools) are most relevant for an image generation agent. Focus on practical recommendations: which plugins to enable, which to skip, and why. Check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Research the best Hermes Agent plugin configuration for an 'image-generator' profile focused on AI image generation, image editing, design workflows, and visual content creation. Consider which plugins (model providers, web search, browser, memory, tools) are most relevant for an image generation agent. Focus on practical recommendations: which plugins to enable, which to skip, and why. Check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
[ASYNC DELEGATION BATCH COMPLETE — deleg_2d07ecd6]
A background fan-out of 8 subagent(s) you dispatched earlier has finished. All ran in parallel and waited on each other; their consolidated results are below. You may have moved on since dispatching — act on these or re-dispatch if things have changed.

Dispatched: 2026-07-13 23:59:18 (11m48s ago)
Context you provided: Clone these 10 repos to /Volumes/this_and_that/projects/multi-repo_0/ using `git clone https://github.com/openclaw/<repo>.git`. Run from /Volumes/this_and_that/projects/multi-repo_0/. Report any failures.
Role: leaf   Model: ?   Total duration: 708.72s

--- ✓ TASK 1/8: Clone 10 OpenClaw repos: openclaw, docs, clawhub, clawgo, homebrew-tap, openclaw.ai, nix-openclaw, nix-openclaw-tools, lobster, hermit  (status=completed, api_calls=3, 356.23s) ---
Context length exceeded (379 tokens). Cannot compress further.

--- ✓ TASK 2/8: Clone 10 OpenClaw repos: clawdinators, openclaw-ansible, casa, spogo, goplaces, voice-community, acpx, community, caclawphony, discrawl  (status=completed, api_calls=4, 525.97s) ---
All 10 repos cloned successfully. Here's the summary:

**Outcome: 10/10 repos cloned successfully, 0 failures.**

All 10 OpenClaw repositories are now present at `/Volumes/this_and_that/projects/multi-repo_0/`:

| # | Repo | Status |
|---|------|--------|
| 1 | clawdinators | ✅ |
| 2 | openclaw-ansible | ✅ |
| 3 | casa | ✅ |
| 4 | spogo | ✅ |
| 5 | goplaces | ✅ |
| 6 | voice-community | ✅ |
| 7 | acpx | ✅ |
| 8 | community | ✅ |
| 9 | caclawphony | ✅ |
| 10 | discrawl | ✅ |

No issues encountered — all clones completed cleanly with exit code 0.

--- ✓ TASK 3/8: Clone 10 OpenClaw repos: slacrawl, crabline, releases, docs, esp-openclaw-node, shellbench, notcrawl, clawsweeper, clownfish, crabpot  (status=completed, api_calls=3, 394.02s) ---
All 10 repos cloned successfully. Here's the summary:

**Result: 10/10 repos cloned with zero failures.**

Cloned to `/Volumes/this_and_that/projects/multi-repo_0/`:

| # | Repo | Status |
|---|------|--------|
| 1 | `slacrawl` | ✅ Cloned |
| 2 | `crabline` | ✅ Cloned |
| 3 | `releases` | ✅ Cloned (15,075 files) |
| 4 | `docs` | ✅ Cloned |
| 5 | `esp-openclaw-node` | ✅ Cloned |
| 6 | `shellbench` | ✅ Cloned (23,628 files) |
| 7 | `notcrawl` | ✅ Cloned |
| 8 | `clawsweeper` | ✅ Cloned |
| 9 | `clownfish` | ✅ Cloned |
| 10 | `crabpot` | ✅ Cloned |

No issues encountered. All 10 repositories are present and fully cloned.

--- ✓ TASK 4/8: Clone 10 OpenClaw repos: wacrawl, plugin-inspector, kitchen-sink, gitcrawl, Kova, cookbook, clawsweeper.bot, clawbox, openclaw-rtt, crawlkit  (status=completed, api_calls=6, 621.05s) ---
Context length exceeded: max compression attempts (3) reached.

--- ✓ TASK 5/8: Clone 10 OpenClaw repos: crawlbar, clawgrit-reports, fs-safe, ask-molty, graincrawl, clickclack, clawdex, proxyline, telecrawl, clawpatch  (status=completed, api_calls=12, 708.31s) ---
All 10 repos cloned successfully. Here's the summary:

**Outcome:** All 10 OpenClaw repositories cloned successfully into `/Volumes/this_and_that/projects/multi-repo_0/`.

**Cloned repos:**
1. ✅ crawlbar
2. ✅ clawgrit-reports (previously cloned)
3. ✅ fs-safe
4. ✅ ask-molty
5. ✅ graincrawl
6. ✅ clickclack
7. ✅ clawdex
8. ✅ proxyline
9. ✅ telecrawl
10. ✅ clawpatch

**Issues:** None. All 10 clones completed with exit code 0.

--- ✓ TASK 6/8: Clone 10 OpenClaw repos: crabfleet, agent-skills, rfcs, rastermill, libopus-wasm, octopool, clawpdf, photoscrawl, clawrouter, imsgcrawl  (status=completed, api_calls=2, 318.11s) ---
All 10 repos cloned successfully. Here's the summary:

**Outcome: 10/10 repos cloned successfully, 0 failures.**

Cloned to `/Volumes/this_and_that/projects/multi-repo_0/`:

| # | Repo | Status |
|---|------|--------|
| 1 | crabfleet | ✅ |
| 2 | agent-skills | ✅ |
| 3 | rfcs | ✅ |
| 4 | rastermill | ✅ |
| 5 | libopus-wasm | ✅ |
| 6 | octopool | ✅ |
| 7 | clawpdf | ✅ |
| 8 | photoscrawl | ✅ |
| 9 | clawrouter | ✅ |
| 10 | imsgcrawl | ✅ |

No issues encountered — all clones completed cleanly with no errors.

--- ✓ TASK 7/8: Clone 10 OpenClaw repos: ffmpeg-wasm, libterminal, clawscan, uirouter, babelfish, crabhelm, turnwire, design-system, reef, AXorcist  (status=completed, api_calls=3, 326.1s) ---
Context length exceeded (435 tokens). Cannot compress further.

--- ✓ TASK 8/8: Clone remaining 8 OpenClaw repos: Peekaboo, Tachikoma, mcporter, Swabble, imsg, gogcli, wacli, songsee, butter.bot  (status=completed, api_calls=3, 355.6s) ---
Context length exceeded (1,277 tokens). Cannot compress further.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Research the best Hermes Agent plugin configuration for an 'image-generator' profile focused on AI image generation, image editing, design workflows, and visual content creation. Consider which plugins (model providers, web search, browser, memory, tools) are most relevant for an image generation agent. Focus on practical recommendations: which plugins to enable, which to skip, and why. Check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
I need to process this. Please help.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Can you find me best Llama 3.3 70B?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Can you execute this project plan --- Attached?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
execute this project plan

--- Attached Context ---

📄 @file:`QUICKSTART.md` (551 tokens)
```markdown
# Quick Start

## 1. Run the Fine-Tuned Model

### MLX (recommended for inference):
```bash
mlx_lm.generate --model ./fused_model --prompt "Your Unity task..."
```

### Ollama:
Create Modelfile in this directory:
```dockerfile
FROM ./gguf/unity-agent-Q6_K.gguf
TEMPLATE "{{ .Prompt }}"
PARAMETER temperature 0.3
PARAMETER num_ctx 4096
```

Then:
```bash
ollama create unity-agent -f Modelfile
ollama serve
```

### LM Studio:
- Open LM Studio → "My Models" → "Add Model"
- Select ./gguf/unity-agent-Q6_K.gguf
- Start server on port 1234

## 2. Connect to Unity Editor

### Option A: Unity-MCP CLI (recommended)
```bash
npx -y @aigamedeveloper/mcp-server
# Connects Claude Desktop, Cursor, or VS Code to Unity Editor
```

### Option B: Custom Editor Window
- Copy `mcp_config/UnityAgentWindow.cs` to your project's `Assets/Editor/`
- Window → Unity Agent → Chat

### Option C: Python client
```bash
python3 mcp_config/unity_mcp_client.py --list
python3 mcp_config/unity_mcp_client.py --tool gameobject-create \
  --args '{"name":"Cube","primitiveType":"Cube"}'
```

## 3. Example Workflows

**Create a scene from scratch:**
> "Create a medieval village with 5 houses, a central fountain, and a player character"

**Fix bugs interactively:**
> "Read the console logs and fix the NullReferenceException in PlayerHealth.cs"

**Profile and optimize:**
> "Profile the game, find the bottleneck, and suggest optimizations"

**Design a system:**
> "Create a ScriptableObject-based inventory system with item pickup, UI display, and save/load"

## 4. Dataset Pipeline (full run)

```bash
# Generate custom datasets
python3 01_build_unity_mcp_dataset.py
python3 02_build_unity_design_patterns_dataset.py

# Download external datasets
python3 05_download_and_merge_datasets.py
python3 06_extract_unity_repos.py
python3 07_integrate_new_datasets.py

# Train + export
python3 03_train_and_export_unity_agent.py
```

## 5. Fine-Tuning Stages

```bash
# Prepare only (merge datasets)
python3 03_train_and_export_unity_agent.py --stage prepare

# Train only (QLoRA)
python3 03_train_and_export_unity_agent.py --stage train

# Fuse adapters and export GGUF
python3 03_train_and_export_unity_agent.py --stage export
```

```

📄 @file:`README.md` (1740 tokens)
```markdown
# Unity Agent Developer — Model Training Pipeline

Build and deploy a Unity game development agent model on Apple Silicon (M5 Pro).

## Quick Start

```bash
# 1. Generate custom datasets
python3 01_build_unity_mcp_dataset.py
python3 02_build_unity_design_patterns_dataset.py

# 2. Download + merge everything (internet needed)
python3 05_download_and_merge_datasets.py       # vishnuOI + function-calling
python3 06_extract_unity_repos.py                # ML-Agents, ECS, uGUI, netcode
python3 07_integrate_new_datasets.py             # Erocal Q&A + Mix & Jam mechanics

# 3. Fine-tune (MLX QLoRA on Apple Silicon)
python3 03_train_and_export_unity_agent.py

# 4. Export to GGUF
python3 03_train_and_export_unity_agent.py --stage export

# 5. Run
ollama create unity-agent -f Modelfile
ollama run unity-agent
```

## Components

| # | Script | What It Does | Output |
|---|---|---|---|
| 01 | `01_build_unity_mcp_dataset.py` | Generates ~128 MCP tool-calling examples from real game-dev tasks | `datasets/unity_mcp_tool_calls_*.jsonl` |
| 02 | `02_build_unity_design_patterns_dataset.py` | Generates ~75 design pattern + bug-fix examples | `datasets/unity_design_patterns_*.jsonl` |
| 03 | `03_train_and_export_unity_agent.py` | MLX QLoRA train → fuse → GGUF export → eval | `adapters/`, `fused_model/`, `gguf/` |
| 04 | `04_setup_unity_mcp_bridge.py` | Sets up Unity-MCP bridge, configs for Claude/Cursor/VS Code | `mcp_config/` |
| 05 | `05_download_and_merge_datasets.py` | Downloads `vishnuOI/unity-dev-instructions` (46k) + `hermes-function-calling-v1` (2k), merges everything | `datasets/train.jsonl` |
| 06 | `06_extract_unity_repos.py` | Extracts C# code from Unity repos: ML-Agents (29), ECS (10), uGUI (10), Netcode (5) | `datasets/unity_repos_*.jsonl` |
| 07 | `07_integrate_new_datasets.py` | Downloads + integrates `Erocal/Unity_ResolvedQuestions` (400 real Q&A) and `MixAndJam/GameMechanics` (32 game mechanic C# codes) | Rebuilds `datasets/{train,valid,test}.jsonl` |

## Dataset Composition (after all merges)

| Source | Rows | Coverage |
|---|---|---|
| `vishnuOI/unity-dev-instructions` | ~42k (after dedup) | Unity API docs, Stack Overflow, GitHub code — scripting, rendering, physics, editor, XR, UI, animation, networking, math, performance, input |
| `NousResearch/hermes-function-calling-v1` | ~1.8k | Generic tool-calling / JSON-mode structured outputs |
| `Erocal/Unity_ResolvedQuestions` | ~400 | Real resolved Q&A — physics, ECS, URP/HDRP, UI Toolkit, editor tooling, platform issues, animation rigging |
| `AmnaHassan/MixAndJam-GameMechanics` | ~32 | Full game mechanic C# implementations: Celeste movement, Splatoon ink, Arkham combat, God of War axe throw, Balatro feel, AC dialogue, Luigi's Mansion ghost capture, Mario Galaxy launch stars, Pikmin gameplay |
| ML-Agents example agents | ~29 | RL agent patterns (Crawler, Walker, FoodCollector, Soccer, Pyramids, WallJump, Worm, Hallway, 3DBall, PushBlock, GridWorld, Sorter) |
| ML-Agents core + sensors | ~5 | Agent.cs lifecycle, ray perception, camera sensor, vector sensor, grid sensor |
| ECS/DOTS samples | ~10 | Jobs, SystemBase, ISystem, IComponentData, IAspect, Baker patterns |
| uGUI | ~10 | Graphic, Image, Text, Button, Slider, Toggle, ScrollRect, Dropdown, InputField, LayoutGroup |
| Netcode for GameObjects | ~5 | NetworkBehaviour, NetworkVariable, RPC, NetworkObject patterns |
| Unity MCP tool calls (custom) | ~128 | Agentic Unity tool use: scene setup, scripting, prefabs, debugging, profiling, testing, shaders, reflection |
| Design patterns (custom) | ~75 | ScriptableObject architecture, Object Pooling, Singleton, State Machine, Observer, Strategy, Command, Factory, Flyweight, Save/Load, ServiceLocator, EventBus |
| **Total (deduped)** | **~45k** | |

### Datasets considered and rejected

| Source | Why skipped |
|---|---|
| `ibranze/codellama_unity3d_v1/v2` | 535 + 2k rows, wrong format, overlaps with vishnuOI |
| `ZorraZabb/code25pile75_shuffle_filtered_unity3d_dedup` | 1M rows of noise — substring "unity3d" filter catches Python, JS, news articles, medical papers |
| `supergoose/buzz_sources_244_unity3d-asset` | 101 rows of git diff YAML for .meta files |
| `robertzeng/unity3ddoc` | 113k rows but shallow/trivial responses (5-15 word generic definitions) |
| `DLMveloper/DLM_DataSet` | 3.6M rows but only ~3,570 "Unity"-tagged, all trivial (variable declarations, basic array ops) |
| `Saibalaji25/unity-csharp-scripts` | Requires login + T&C agreement, unknown quality, 11 downloads/mo |

## Training Configuration (M5 Pro 64GB)

```
Model:      Qwen2.5-Coder-32B-Instruct (4-bit QLoRA)
Batch size: 1
Layers:     16/64 (25% LoRA)
LR:         1e-4
Max seq:    4096
Iters:      2000
Peak mem:   ~25GB
Duration:   ~45-60 min
```

## GGUF Output Quantization

| Type | Size | Best For |
|---|---|---|
| Q4_K_M | ~20 GB | 24GB+ systems |
| Q5_K_M | ~23 GB | 32GB+ systems |
| **Q6_K** | **~27 GB** | **M5 Pro 64GB (recommended)** |
| Q8_0 | ~34 GB | 64GB+ systems (max quality) |

## Unity-MCP Bridge

Run the bridge setup to configure connections to Claude Desktop, Cursor, VS Code, or the included Python client:

```bash
python3 04_setup_unity_mcp_bridge.py
```

Then connect any MCP-compatible AI client (Claude Desktop, Cursor, VS Code with Cline/Copilot, Codex) to the Unity Editor.

## Project Structure

```
unity-agent-training/
├── 01_build_unity_mcp_dataset.py            # MCP tool-calling dataset
├── 02_build_unity_design_patterns_dataset.py  # Design patterns dataset
├── 03_train_and_export_unity_agent.py        # MLX training + GGUF export
├── 04_setup_unity_mcp_bridge.py              # Unity-MCP integration
├── 05_download_and_merge_datasets.py         # HF dataset download + merge
├── 06_extract_unity_repos.py                 # Unity org repo C# extraction
├── 07_integrate_new_datasets.py              # Newly discovered datasets
├── Modelfile                                 # Ollama model definition
├── QUICKSTART.md                             # Quick start guide
├── datasets/
│   ├── unity_mcp_tool_calls_{train,valid,test}.jsonl
│   ├── unity_design_patterns_{train,valid,test}.jsonl
│   ├── unity_api_instructions.jsonl          # (downloaded)
│   ├── unity_repos_{train,valid,test}.jsonl  # (from repo extraction)
│   ├── function_calling_full.jsonl           # (downloaded)
│   ├── train.jsonl                           # Merged MLX-ready
│   ├── valid.jsonl                           # Merged MLX-ready
│   └── test.jsonl                            # Merged MLX-ready
├── adapters/                                 # LoRA adapters (after training)
├── fused_model/                              # Fused MLX model
├── gguf/                                     # GGUF quantized models
└── mcp_config/                               # Bridge configuration
    ├── claude_desktop_config.json
    ├── vscode_mcp.json
    ├── unity_mcp_server_config.json
    ├── unity_mcp_client.py
    └── UnityAgentWindow.cs
```

```<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
i need to know if you are making a GGUF model or MLX model?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
[ASYNC DELEGATION BATCH COMPLETE — deleg_2d07ecd6]
A background fan-out of 8 subagent(s) you dispatched earlier has finished. All ran in parallel and waited on each other; their consolidated results are below. You may have moved on since dispatching — act on these or re-dispatch if things have changed.

Dispatched: 2026-07-13 23:59:18 (11m48s ago)
Context you provided: Clone these 10 repos to /Volumes/this_and_that/projects/multi-repo_0/ using `git clone https://github.com/openclaw/<repo>.git`. Run from /Volumes/this_and_that/projects/multi-repo_0/. Report any failures.
Role: leaf   Model: ?   Total duration: 708.72s

--- ✓ TASK 1/8: Clone 10 OpenClaw repos: openclaw, docs, clawhub, clawgo, homebrew-tap, openclaw.ai, nix-openclaw, nix-openclaw-tools, lobster, hermit  (status=completed, api_calls=3, 356.23s) ---
Context length exceeded (379 tokens). Cannot compress further.

--- ✓ TASK 2/8: Clone 10 OpenClaw repos: clawdinators, openclaw-ansible, casa, spogo, goplaces, voice-community, acpx, community, caclawphony, discrawl  (status=completed, api_calls=4, 525.97s) ---
All 10 repos cloned successfully. Here's the summary:

**Outcome: 10/10 repos cloned successfully, 0 failures.**

All 10 OpenClaw repositories are now present at `/Volumes/this_and_that/projects/multi-repo_0/`:

| # | Repo | Status |
|---|------|--------|
| 1 | clawdinators | ✅ |
| 2 | openclaw-ansible | ✅ |
| 3 | casa | ✅ |
| 4 | spogo | ✅ |
| 5 | goplaces | ✅ |
| 6 | voice-community | ✅ |
| 7 | acpx | ✅ |
| 8 | community | ✅ |
| 9 | caclawphony | ✅ |
| 10 | discrawl | ✅ |

No issues encountered — all clones completed cleanly with exit code 0.

--- ✓ TASK 3/8: Clone 10 OpenClaw repos: slacrawl, crabline, releases, docs, esp-openclaw-node, shellbench, notcrawl, clawsweeper, clownfish, crabpot  (status=completed, api_calls=3, 394.02s) ---
All 10 repos cloned successfully. Here's the summary:

**Result: 10/10 repos cloned with zero failures.**

Cloned to `/Volumes/this_and_that/projects/multi-repo_0/`:

| # | Repo | Status |
|---|------|--------|
| 1 | `slacrawl` | ✅ Cloned |
| 2 | `crabline` | ✅ Cloned |
| 3 | `releases` | ✅ Cloned (15,075 files) |
| 4 | `docs` | ✅ Cloned |
| 5 | `esp-openclaw-node` | ✅ Cloned |
| 6 | `shellbench` | ✅ Cloned (23,628 files) |
| 7 | `notcrawl` | ✅ Cloned |
| 8 | `clawsweeper` | ✅ Cloned |
| 9 | `clownfish` | ✅ Cloned |
| 10 | `crabpot` | ✅ Cloned |

No issues encountered. All 10 repositories are present and fully cloned.

--- ✓ TASK 4/8: Clone 10 OpenClaw repos: wacrawl, plugin-inspector, kitchen-sink, gitcrawl, Kova, cookbook, clawsweeper.bot, clawbox, openclaw-rtt, crawlkit  (status=completed, api_calls=6, 621.05s) ---
Context length exceeded: max compression attempts (3) reached.

--- ✓ TASK 5/8: Clone 10 OpenClaw repos: crawlbar, clawgrit-reports, fs-safe, ask-molty, graincrawl, clickclack, clawdex, proxyline, telecrawl, clawpatch  (status=completed, api_calls=12, 708.31s) ---
All 10 repos cloned successfully. Here's the summary:

**Outcome:** All 10 OpenClaw repositories cloned successfully into `/Volumes/this_and_that/projects/multi-repo_0/`.

**Cloned repos:**
1. ✅ crawlbar
2. ✅ clawgrit-reports (previously cloned)
3. ✅ fs-safe
4. ✅ ask-molty
5. ✅ graincrawl
6. ✅ clickclack
7. ✅ clawdex
8. ✅ proxyline
9. ✅ telecrawl
10. ✅ clawpatch

**Issues:** None. All 10 clones completed with exit code 0.

--- ✓ TASK 6/8: Clone 10 OpenClaw repos: crabfleet, agent-skills, rfcs, rastermill, libopus-wasm, octopool, clawpdf, photoscrawl, clawrouter, imsgcrawl  (status=completed, api_calls=2, 318.11s) ---
All 10 repos cloned successfully. Here's the summary:

**Outcome: 10/10 repos cloned successfully, 0 failures.**

Cloned to `/Volumes/this_and_that/projects/multi-repo_0/`:

| # | Repo | Status |
|---|------|--------|
| 1 | crabfleet | ✅ |
| 2 | agent-skills | ✅ |
| 3 | rfcs | ✅ |
| 4 | rastermill | ✅ |
| 5 | libopus-wasm | ✅ |
| 6 | octopool | ✅ |
| 7 | clawpdf | ✅ |
| 8 | photoscrawl | ✅ |
| 9 | clawrouter | ✅ |
| 10 | imsgcrawl | ✅ |

No issues encountered — all clones completed cleanly with no errors.

--- ✓ TASK 7/8: Clone 10 OpenClaw repos: ffmpeg-wasm, libterminal, clawscan, uirouter, babelfish, crabhelm, turnwire, design-system, reef, AXorcist  (status=completed, api_calls=3, 326.1s) ---
Context length exceeded (435 tokens). Cannot compress further.

--- ✓ TASK 8/8: Clone remaining 8 OpenClaw repos: Peekaboo, Tachikoma, mcporter, Swabble, imsg, gogcli, wacli, songsee, butter.bot  (status=completed, api_calls=3, 355.6s) ---
Context length exceeded (1,277 tokens). Cannot compress further.<|im_end|>


Prompt:
<|im_start|>system
You are in General Assistant profile. Use the most efficient tool for each request.
<|im_end|>
<|im_start|>user
Research the best Hermes Agent plugin configuration for a 'cybersec' profile focused on security auditing, penetration testing, vulnerability research, and security analysis. Consider which plugins (model providers, web search, browser, memory, tools) are most relevant for a security-focused agent. Focus on practical recommendations: which plugins to enable, which to skip, and why. Check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
I need to process proc_0a497b8b5935 exited (exit code -15,. Please help.<|im_end|>


Prompt:
<|im_start|>system
You are in Developer profile. Prioritize code execution, file editing, and terminal access.
<|im_end|>
<|im_start|>user
Research the best Hermes Agent plugin configuration for a 'cybersec' profile focused on security auditing, penetration testing, vulnerability research, and security analysis. Consider which plugins (model providers, web search, browser, memory, tools) are most relevant for a security-focused agent. Focus on practical recommendations: which plugins to enable, which to skip, and why. Check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Can you check https://hermes-agent.nousresearch.com/docs for any profile-specific guidance.?<|im_end|>


Prompt:
<|im_start|>system
You are an AI assistant inside Hermes Desktop. Use tools when appropriate. Think before acting, verify after each tool call.
<|im_end|>
<|im_start|>user
Can you process this?<|im_end|>