leapeto/mindcube-latent-ckpts

MindCube latent-reasoning checkpoints

LoRA adapters for Qwen/Qwen2.5-VL / Qwen2.5-VL-3B-Instruct that reason over MindCube spatial questions in a compressed latent vocabulary — the model emits <latent_start> <lat_…> <latent_end> <answer>X</answer> — instead of a natural-language chain of thought. Each <lat_i> is one BPE-merged unit standing for several base tokens, so the reasoning decodes in far fewer steps at no accuracy cost.

Training traces (text): leapeto/mindcube-latent-data. Images: Inevitablevalor/MindCube data.zip.

Checkpoints

Frozen-3B reference ≈ 37.8%; uncompressed full-CoT SFT baseline (plain_3b_ep5) = 46.6%. Accuracy is flat at ~49–51% across the whole 1.5×–4.05× compression sweep — compression is free down to the merge-table ceiling. Tinybench = MindCube's 1,050-question eval split, greedy two-stage decode.

adapter	base	target compression	M (<lat_i>)	median decode tok	tinybench acc
plain_3b_ep5	3B	1.0× (uncompressed full-CoT)	—	321	46.6%
c1.5mn_3b_ep5	3B	1.5×	1,961	207	49.7%
c2.0mn_3b_ep5	3B	2.0×	1,983	145	51.1%
c2.5mn_3b_ep5	3B	2.5×	2,082	124	51.0%
c3.0mn_3b_ep5	3B	3.0×	2,294	102	47.6%
c3.5mn_3b_ep5	3B	3.5×	3,011	88	50.1%
c2.0mn_7b_ep4	7B	2.0×	1,983	144	49.6%
ffrsn_3b_ep5	3B	2.0× (free-form reasoning target, no map)	—	33	51.9%
c4.0mn2_3b_ep5	3B	4.0×	9,081	76	49.3%
c4.5mn2_3b_ep5	3B	4.05× (merge ceiling)	11,895	73	49.1%
c2.0mn2_3b_ep5	3B	2.0× (regen-corpus anchor)	1,993	150	49.1%

The mn2 checkpoints are the high-compression extension: same recipe, trained on a regenerated (self-distilled) copy of the trace corpus. c2.0mn2 reproduces the original c2.0mn within noise (49.1% vs 51.1%) and exists only to confirm the corpora are comparable. Targets 4.5× and 5.0× both collapse to the same 4.05× vocabulary — the merge table runs out of recurring structure — so c4.5mn2 is the ceiling.

Usage

Each adapter dir bundles its tokenizer (defines the <lat_i> tokens) and — for the mn2 checkpoints — a latent_vocab.json sidecar mapping every <lat_i> back to its base-token span (needed only to expand the latent output into readable text or to retrain; plain inference does not need it). Decode in two greedy stages with a logit mask:

1. Latent — from the prompt + <latent_start>, restrict next-token logits to the <lat_i> units ∪ <latent_end>; stop on <latent_end> (or a cap ~400).
2. Answer — block the <lat_i> units and <latent_end>; decode <answer>X</answer>.

from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from peft import PeftModel
import torch

name = "c4.0mn2_3b_ep5"
proc = AutoProcessor.from_pretrained(f"leapeto/mindcube-latent-ckpts/{name}")
base = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-3B-Instruct", dtype=torch.bfloat16, device_map="cuda")
base.resize_token_embeddings(len(proc.tokenizer))     # absorb the added <lat_i> units
model = PeftModel.from_pretrained(base, f"leapeto/mindcube-latent-ckpts/{name}").eval()
# then the two-stage masked greedy decode above.

M-RoPE gotcha: in a manual KV-cache decode loop, pass cache_position explicitly every step. Qwen2.5-VL derives decode positions from cache_position + rope_deltas; omitting it feeds every generated token at position 0 and produces repetition/garbage (transformers.generate / vLLM handle this automatically).

Prompt format: MindCube aug_cgmap_ffr_out. Benchmark: MindCube (arXiv:2506.21458).