Reproducible original-vs-1019 agent evaluation
Thanks for releasing 1019 and the base checkpoint. At pinned revisions, the earlier chat release and 1019 publish eight byte-identical setup, tokenizer, loader and weight-index files. Their 54 safetensors shard paths and sizes match, while all 54 LFS SHA-256 hashes differ in Hugging Face's earlier and 1019 revision metadata. The separate base checkpoint differs in prompt and setup files, including its EOS ID and chat template.
For the model card's reported AutomationBench-AA increase from 62% to 68%, could you share paired per-task results or an evaluation recipe with the harness commit, prompt serialization, sampling, runtime/tool parser, context and output budgets, and failure accounting? Matching setup files make the earlier chat checkpoint the cleaner control, but shard hashes cannot explain the score change. Our source-pinned audit and reproduction code show the exact file comparison.