Benchmark & generation scripts

Harnesses used to benchmark and generate from the Quazimoto (Wheeler/Positronic) and SpikeWhale (Byrne) model families.

Benchmark

file	what it does
`run_eval.py`	Main benchmark. WikiText-2 `byte_ppl`, BLiMP (12 paradigms x150), MC suite (arc_easy/challenge, hellaswag, winogrande, piqa, openbookqa, boolq -> acc & acc_norm), and ArithMark-3.0. Handles both families. Self-contained.
`dpo_eval_quazimoto.py`	Held-out DPO preference accuracy (UltraFeedback `test_prefs`) for a Quazimoto policy vs frozen reference.
`dpo_eval_spikewhale.py`	Same, for a SpikeWhale policy. Needs `sw_common.py`.
`sw_common.py`	Shared SpikeWhale helpers (loader, ChatML renderer, bucketed-pad seq log-prob).

python run_eval.py --family {quazimoto|spikewhale} \
  --code_dir <model-repo-dir> --ckpt <ckpt.pt-or-dir> \
  --name <label> --out out/<label>.json

Generation

file	family	modes
`gen_quazimoto_single.py`	Quazimoto (Wheeler/Positronic)	5 single-turn prompts
`gen_quazimoto_multiturn.py`	Quazimoto	2 scripted multi-turn conversations
`gen_spikewhale.py`	SpikeWhale (Byrne)	`--mode single` or `--mode multi`
`infer_v2.py`	—	sampler used by `gen_spikewhale.py`

# Quazimoto (run from the model repo dir so `generate.py`/`model.py` import)
python gen_quazimoto_single.py   --ckpt <ckpt.pt> --device cuda
python gen_quazimoto_multiturn.py --ckpt <ckpt.pt> --device cuda

# SpikeWhale (run from the Byrne Mark2 dir so `model_v2.py`/`config.py` import)
python gen_spikewhale.py --ckpt <ckpt.pt-or-dir> --device cuda --mode single
python gen_spikewhale.py --ckpt <ckpt.pt-or-dir> --device cuda --mode multi

Dependencies / how to run

run_eval.py is standalone. The generation and DPO-eval scripts load models via each repo's OWN code, so run them from inside (or with on the path) the model repo:

Quazimoto scripts import generate.py / model.py (Wheeler/Positronic repo).
SpikeWhale scripts import model_v2.py / config.py / spike_tokenizer.py (Byrne Mark2/); gen_spikewhale.py also uses infer_v2.py (copied here) and sw_common.py.

The tokenizer (tokenizer.json + spike_tokenizer.py) lives in the model repos and is shared across all these models (SpikeWhale byte-level, vocab 16512).

Notes

ArithMark target is AxiomicLabs/ArithMark-3.0 (top of run_eval.py).
Datasets stream from HuggingFace; datasets/pyarrow import before torch/CUDA (Windows ordering).
DPO scripts pad sequence lengths to multiples of 128 to avoid CUDA allocator fragmentation on Windows (no expandable_segments there).