Spaces:

---

FlameF0X

/

Micro-ARC

Like 2

Running

marc.py currently invokes HF generation once per puzzle. This change pretokenizes the selected run, batches similarly sized prompts (default 32), and restores responses to their original puzzle order before scoring. It is self-contained: only marc.py changes, with no dependency on another benchmark toolkit.

• Cached input IDs, left padding, KV-cached generation, and last-position logits when the model explicitly supports them.
• CUDA OOM backoff retries only the unsuccessful batch, down to batch size one.
• Live completed-puzzle/decode-token counts, elapsed time, throughput, ETA, and periodic prefill/decode heartbeats. After-EOS padding is excluded from counts and parser input; EOS and <think> token spellings remain preserved.
• --batch-size, --max-model-len, --dtype, --progress-interval, and --quiet. --batch-size 1 provides the single-item comparison path. CPU defaults to FP32; CUDA retains BF16.
• Full prompts are validated against the context budget instead of silently dropping puzzle demonstrations. Existing HF base/chat prompts, multi-sample decoding, multi-model buckets, and OpenAI-compatible inference remain supported.

Task generators, seed construction, demonstrations, prompt formatting, grid parsing, and _score are unchanged. The public result files, UI, and README are untouched. The pretokenization cache is in-memory for each model/run, not a persistent disk cache.

Example:

python marc.py --hf HuggingFaceTB/SmolLM2-360M --batch-size 32
python marc.py --hf YOUR_MODEL --device cpu --dtype float32 --batch-size 8

Validation against base commit 4ed8c708aa3e9665a9d9c81f2e3bc586cf60e14e:

• python marc.py --selftest: passed the existing parsing checks and 4,800 seeded leak-invariant cases.
• AST comparison confirms the generators, formatters, parser, and scorer are unchanged; all 320 default tasks produce the same 16 aggregate scores with a deterministic reference model.
• Seven local review tests pass, including tiny GPT-2 CPU greedy parity with the old single-prompt code, batch-size-one parity, chat-tokenization parity, sampled multi-return shapes, cache reuse, no-truncation checks, output-count validation, and simulated CUDA OOM/order/EOS handling.

Batched floating-point inference may differ near logit ties, and sampling consumes RNG in a different order. Exact sampled-output parity is not claimed. No GPU throughput claim is made; real GPU benchmarking remains useful maintainer validation.