| # Benchmark & generation scripts |
|
|
| Harnesses used to benchmark and generate from the Quazimoto (Wheeler/Positronic) |
| and SpikeWhale (Byrne) model families. |
|
|
| ## Benchmark |
|
|
| | file | what it does | |
| |---|---| |
| | `run_eval.py` | **Main benchmark.** WikiText-2 `byte_ppl`, BLiMP (12 paradigms x150), MC suite (arc_easy/challenge, hellaswag, winogrande, piqa, openbookqa, boolq -> acc & acc_norm), and ArithMark-3.0. Handles **both** families. Self-contained. | |
| | `dpo_eval_quazimoto.py` | Held-out DPO preference accuracy (UltraFeedback `test_prefs`) for a Quazimoto policy vs frozen reference. | |
| | `dpo_eval_spikewhale.py` | Same, for a SpikeWhale policy. Needs `sw_common.py`. | |
| | `sw_common.py` | Shared SpikeWhale helpers (loader, ChatML renderer, bucketed-pad seq log-prob). | |
|
|
| ```bash |
| python run_eval.py --family {quazimoto|spikewhale} \ |
| --code_dir <model-repo-dir> --ckpt <ckpt.pt-or-dir> \ |
| --name <label> --out out/<label>.json |
| ``` |
|
|
| ## Generation |
|
|
| | file | family | modes | |
| |---|---|---| |
| | `gen_quazimoto_single.py` | Quazimoto (Wheeler/Positronic) | 5 single-turn prompts | |
| | `gen_quazimoto_multiturn.py` | Quazimoto | 2 scripted multi-turn conversations | |
| | `gen_spikewhale.py` | SpikeWhale (Byrne) | `--mode single` or `--mode multi` | |
| | `infer_v2.py` | — | sampler used by `gen_spikewhale.py` | |
|
|
| ```bash |
| # Quazimoto (run from the model repo dir so `generate.py`/`model.py` import) |
| python gen_quazimoto_single.py --ckpt <ckpt.pt> --device cuda |
| python gen_quazimoto_multiturn.py --ckpt <ckpt.pt> --device cuda |
| |
| # SpikeWhale (run from the Byrne Mark2 dir so `model_v2.py`/`config.py` import) |
| python gen_spikewhale.py --ckpt <ckpt.pt-or-dir> --device cuda --mode single |
| python gen_spikewhale.py --ckpt <ckpt.pt-or-dir> --device cuda --mode multi |
| ``` |
|
|
| ## Dependencies / how to run |
|
|
| `run_eval.py` is standalone. The generation and DPO-eval scripts load models via |
| each repo's OWN code, so run them **from inside (or with on the path) the model |
| repo**: |
| - **Quazimoto** scripts import `generate.py` / `model.py` (Wheeler/Positronic repo). |
| - **SpikeWhale** scripts import `model_v2.py` / `config.py` / `spike_tokenizer.py` |
| (Byrne `Mark2/`); `gen_spikewhale.py` also uses `infer_v2.py` (copied here) and `sw_common.py`. |
|
|
| The tokenizer (`tokenizer.json` + `spike_tokenizer.py`) lives in the model repos and |
| is shared across all these models (SpikeWhale byte-level, vocab 16512). |
|
|
| ## Notes |
|
|
| - ArithMark target is `AxiomicLabs/ArithMark-3.0` (top of `run_eval.py`). |
| - Datasets stream from HuggingFace; `datasets`/`pyarrow` import before torch/CUDA (Windows ordering). |
| - DPO scripts pad sequence lengths to multiples of 128 to avoid CUDA allocator |
| fragmentation on Windows (no `expandable_segments` there). |
|
|
