Patch
#2
by wop - opened
- Script patched script.py,
- extract_answer now strips reasoning before scoring: it keeps only the text after the last (also handles , and the case where the chat template opens the tag in the prompt so the generation contains only a bare ). An unclosed โ generation truncated mid-reasoning โ scores as an empty answer rather than grading reasoning text. Verified with 7 test cases, all passing.
- --max-new-tokens help text and the module docstring now say to use 2048+ for reasoning models, and note that mid-tier is log-likelihood-only so thinking never runs there.
Revision capture flipped, per the review: model.config._commit_hash (the snapshot from_pretrained actually loaded โ no second Hub lookup) is now primary, and HfApi().model_info().sha is only a fallback for transformers versions that don't record it. Local checkpoints stay None. The comment now explains the race the reviewer described: re-resolving main after load can pin commit B while the scores came from A.
Docs updated
- index.html โ added a reasoning-models paragraph to "How to evaluate"; replaced the stale eval_source rule with the automatic provenance pinning (model_revision + script_sha256, eval link goes in the PR description); updated the entry-schema snippet to show the new fields instead of eval_source.
- README.md โ added "Reasoning / CoT models" and "Provenance" sections mirroring the same points.
ereniko changed pull request status to merged
wop deleted the
refs/pr/2 ref