README.md

---
license: mit
library_name: pytorch
tags:
- reinforcement-learning
- game-ai
- slay-the-spire
- reinforce
- policy
---

# sts-rl-agent — a learned non-combat policy for Slay the Spire

[English](#sts-rl-agent--a-learned-non-combat-policy-for-slay-the-spire) | [中文](#中文说明)

To our knowledge, the **first working learned policy published for [sts_lightspeed](https://github.com/gamerpuppy/sts_lightspeed)** — the fast C++ Slay the Spire simulator, which ships a 412-dim neural-network observation interface (`NNInterface`) but no trained weights.

A small MLP (~100k params, `[128,128]`, trained from scratch with REINFORCE) makes **all non-combat decisions** — map pathing, card rewards, shops, campfires, events — while the simulator's built-in MCTS plays combat.

## Headline result

Same MCTS combat, same 50 held-out seeds, A0 Ironclad — only the non-combat "brain" differs:

| non-combat decisions | combat | avg floor | win rate |
|---|---|---|---|
| stock bot heuristics (map = random) | MCTS @50000 | 31.2 | 6% |
| **this model** | MCTS @50000 | **42.5** | **14%** |

The learned non-combat layer is worth **~11 floors** over the stock bot: its biggest weakness was never combat, it was walking the map at random.

## Files

| file | what |
|---|---|
| `armG_model_G128x128_15k.pt` | **the non-combat policy** behind the headline number (15k games) |
| `armG_model_G128x128.pt` | earlier 8k-game checkpoint |
| `armS_card_vocab.json` | card vocabulary (required to encode candidates) |
| `armB_model_B256x256.pt` | combat behavior-cloning model — **negative result** (floor ~12 vs teacher 23) |
| `armB_model_VAL256x256.pt` | combat value net — negative result (1-ply lookahead: floor ~8) |
| `armB_model_ATTN_d64L2.pt` | combat attention model — negative result (floor ~14) |

The combat models are published on purpose: six different attempts to distill MCTS combat into a feed-forward network all failed the same way (imitation caps at 0.44 train accuracy — the MCTS teacher effectively sees the future draw order; a one-frame policy can't). *Judgment*-type decisions compress into small networks easily; *planning*-type decisions resist.

## Usage

Input is `obs(412) ⊕ candidate-descriptor(368)` → scalar score per candidate; pick the argmax. You need the patched simulator and the encoding code — **full code, sim patch, training scripts and eval protocol: [github.com/valiant-wjl/sts-rl-agent](https://github.com/valiant-wjl/sts-rl-agent)**.

```python
import torch
from agent.armG_train import Scorer, build_choices, obs_vec   # from the GitHub repo
net = Scorer((128, 128))
net.load_state_dict(torch.load("armG_model_G128x128_15k.pt", weights_only=True))
```

## Limitations

- A0 (lowest difficulty), Ironclad only (the simulator only fully implements Ironclad).
- Combat is still search (MCTS), not learned.
- Single-run numbers on 50 fixed seeds, no confidence intervals.

Slay the Spire is a trademark of Mega Crit Games; this is an unaffiliated research project on a clean-room simulator (MIT).


---

# 中文说明

据我们所知,这是**第一个针对 [sts_lightspeed](https://github.com/gamerpuppy/sts_lightspeed) 神经网络接口公开发布的、能用的学习型策略**——这个快速 C++《杀戮尖塔》模拟器自带 412 维观测接口(`NNInterface`),但作者从未公开过训好的权重。

一个从零训练的小 MLP(约 10 万参数,`[128,128]`,REINFORCE)做**全部非战斗决策**——地图选路、奖励选卡、商店、篝火、事件;战斗由模拟器内置的 MCTS 执行。

## 核心结果

同样的 MCTS 战斗、同样的 50 个留出关卡(A0 铁甲),只换"非战斗的脑子":

| 非战斗决策 | 战斗 | 平均楼层 | 通关率 |
|---|---|---|---|
| 原生 bot 启发式(地图=随机) | MCTS @50000 | 31.2 | 6% |
| **本模型** | MCTS @50000 | **42.5** | **14%** |

学习型非战斗层比原生 bot **高出约 11 层**——它最大的软肋从来不是打牌,而是选路基本靠随机。

## 文件说明

| 文件 | 说明 |
|---|---|
| `armG_model_G128x128_15k.pt` | **核心结果背后的非战斗策略**(1.5 万局训练) |
| `armG_model_G128x128.pt` | 更早的 8 千局 checkpoint |
| `armS_card_vocab.json` | 卡牌词表(编码候选项必需) |
| `armB_model_B256x256.pt` | 战斗行为克隆模型——**负结果**(实战 ~12 层 vs 老师 23) |
| `armB_model_VAL256x256.pt` | 战斗价值网——负结果(1 步前瞻只有 ~8 层) |
| `armB_model_ATTN_d64L2.pt` | 战斗注意力模型——负结果(~14 层) |

战斗模型是**有意公开的负结果**:六种把 MCTS 战斗蒸馏进前馈网络的方法全部以同样方式失败(模仿训练准确率卡死 0.44——MCTS 模拟用的是真实未来抽牌顺序,老师"看得见未来",只看当前一帧的策略学不像)。**判断型决策容易压进小网络;规划型决策会抵抗。**

## 使用

输入 = `obs(412) ⊕ 候选描述符(368)` → 每个候选一个分,取 argmax。需要打过 patch 的模拟器和编码代码——**完整代码、sim patch、训练脚本、评测协议见 [github.com/valiant-wjl/sts-rl-agent](https://github.com/valiant-wjl/sts-rl-agent)**(含中文 README 和架构图)。

## 局限

- 只测了 A0(最低难度)、只有铁甲(模拟器只完整实现了铁甲);
- 战斗仍是搜索(MCTS),不是学出来的;
- 50 个固定 seed 的单次结果,无置信区间。

《杀戮尖塔》(Slay the Spire)是 Mega Crit Games 的商标;本项目是基于净室模拟器的非官方研究项目(MIT)。