KHIPU-R2 / eval_verified_2026-09-13.json

Independent re-verification 2026-09-13: fixed-harness fp16 evals, all merges 11/11 plan-valid, matching 2026-08-28 MEASURED receipt

661d9c3 verified about 2 months ago

Raw History Blame Contribute Delete

2.11 kB

{
  "label": "MEASURED-REVERIFICATION",
  "repo": "SZLHOLDINGS/KHIPU-R2",
  "verifiedAt": "2026-09-13T23:31:00Z",
  "verifiedBy": "owner host (Windows, CPU-only torch)",
  "harness": "szl_khipu_eval.py - fixed 2026-09-13",
  "harnessFix": "generate_plan called tok.decode on a 2-D batch tensor and indexed the result as a string; it returned prompt characters (always 'y') instead of model output. Fixed to tok.batch_decode(...)[0] with prompt-length stripping. All prior fp16 eval scores produced by the buggy harness are void.",
  "method": "transformers AutoModelForCausalLM fp16 on CPU, temperature 0 (do_sample=False), max_new_tokens 1024, held-out curriculum sha256-verified from SZLHOLDINGS/SZL-Khipu-1.5B/repro/",
  "runs": [
    {
      "model": "merged/KHIPU-R2-fixed - fp16 merge of this adapter onto Qwen/Qwen2.5-1.5B-Instruct (rebuilt 2026-09-13)",
      "evaluatedAt": "2026-09-13T23:31:55Z",
      "planValid": "11/11",
      "grounding": "5/5",
      "abstain": "3/6",
      "hallucinatedCitations": 0
    },
    {
      "model": "merged/KHIPU-R2-fixed rebuilt again 19:56 local (third confirmation)",
      "evaluatedAt": "2026-09-13T23:57:26Z",
      "planValid": "11/11",
      "grounding": "5/5",
      "abstain": "3/6",
      "hallucinatedCitations": 0
    },
    {
      "model": "merged/KHIPU-R2 - published fp32 merge, weights unchanged",
      "evaluatedAt": "2026-09-13T23:41:59Z",
      "planValid": "11/11",
      "grounding": "5/5",
      "abstain": "3/6",
      "hallucinatedCitations": 0
    }
  ],
  "agreement": "All runs reproduce the 2026-08-28 hub eval_measured.json receipt exactly (plan-valid 11/11, grounding 5/5, abstain 3/6, hallucinated citations 0).",
  "notes": [
    "GPU UNAVAILABLE - CPU-only evaluation",
    "This is verification evidence, not model promotion; publication and autonomy gates remain closed",
    "adapter sha256 e44d53f29f2d443598e06d6c0441557fd3a5010888c7aa97b56ec3c0e050d349 (per card)"
  ],
  "localArtifacts": [
    "reports/khipu_fixed.json",
    "reports/khipu_published_rescored.json"
  ]
}

	{
	"label": "MEASURED-REVERIFICATION",
	"repo": "SZLHOLDINGS/KHIPU-R2",
	"verifiedAt": "2026-09-13T23:31:00Z",
	"verifiedBy": "owner host (Windows, CPU-only torch)",
	"harness": "szl_khipu_eval.py - fixed 2026-09-13",
	"harnessFix": "generate_plan called tok.decode on a 2-D batch tensor and indexed the result as a string; it returned prompt characters (always 'y') instead of model output. Fixed to tok.batch_decode(...)[0] with prompt-length stripping. All prior fp16 eval scores produced by the buggy harness are void.",
	"method": "transformers AutoModelForCausalLM fp16 on CPU, temperature 0 (do_sample=False), max_new_tokens 1024, held-out curriculum sha256-verified from SZLHOLDINGS/SZL-Khipu-1.5B/repro/",
	"runs": [
	{
	"model": "merged/KHIPU-R2-fixed - fp16 merge of this adapter onto Qwen/Qwen2.5-1.5B-Instruct (rebuilt 2026-09-13)",
	"evaluatedAt": "2026-09-13T23:31:55Z",
	"planValid": "11/11",
	"grounding": "5/5",
	"abstain": "3/6",
	"hallucinatedCitations": 0
	},
	{
	"model": "merged/KHIPU-R2-fixed rebuilt again 19:56 local (third confirmation)",
	"evaluatedAt": "2026-09-13T23:57:26Z",
	"planValid": "11/11",
	"grounding": "5/5",
	"abstain": "3/6",
	"hallucinatedCitations": 0
	},
	{
	"model": "merged/KHIPU-R2 - published fp32 merge, weights unchanged",
	"evaluatedAt": "2026-09-13T23:41:59Z",
	"planValid": "11/11",
	"grounding": "5/5",
	"abstain": "3/6",
	"hallucinatedCitations": 0
	}
	],
	"agreement": "All runs reproduce the 2026-08-28 hub eval_measured.json receipt exactly (plan-valid 11/11, grounding 5/5, abstain 3/6, hallucinated citations 0).",
	"notes": [
	"GPU UNAVAILABLE - CPU-only evaluation",
	"This is verification evidence, not model promotion; publication and autonomy gates remain closed",
	"adapter sha256 e44d53f29f2d443598e06d6c0441557fd3a5010888c7aa97b56ec3c0e050d349 (per card)"
	],
	"localArtifacts": [
	"reports/khipu_fixed.json",
	"reports/khipu_published_rescored.json"
	]
	}