Upload tr3/heal/llama2-13b-base/eval_protocol_llama2_13b_base.json with huggingface_hub

tr3/heal/llama2-13b-base/eval_protocol_llama2_13b_base.json

@@ -0,0 +1,55 @@
+{
+  "model_id": "NousResearch/Llama-2-13b-hf",
+  "model_revision": "4b54e9fdd2ebd8db0901d37b98d3a53bbfaa4503",
+  "lm_eval_revision": "0.4.12",
+  "tasks": {
+    "mmlu": "mmlu",
+    "hellaswag": "hellaswag",
+    "boolq": "boolq",
+    "triviaqa": "triviaqa",
+    "gsm8k": "gsm8k"
+  },
+  "num_fewshot": {
+    "mmlu": 5,
+    "hellaswag": 10,
+    "boolq": 0,
+    "triviaqa": 5,
+    "gsm8k": 5
+  },
+  "apply_chat_template": false,
+  "batch_size": "auto",
+  "dtype": "float16",
+  "seed": 1234,
+  "score_scale": "fraction",
+  "result_parser": {
+    "mmlu": "acc,none",
+    "hellaswag": "acc_norm,none",
+    "boolq": "acc,none",
+    "triviaqa": "exact_match,remove_whitespace",
+    "gsm8k": "exact_match,strict-match"
+  },
+  "hardware": {
+    "gpus_max": 4,
+    "gpu_model": "NVIDIA GB200 (Grace-Blackwell NVL4, sm_100)",
+    "note": "RIKYU compute node, 4x GB200 with 184GiB each. NOT the hardware the 7B results were produced on (that was 4x RTX A6000, sm_86). Numerically irrelevant for fp16 eval but recorded because the software stack differs too -- see torch_deviation below."
+  },
+  "torch_deviation": {
+    "locked": "torch==2.4.1+cu121",
+    "actual": "torch==2.7.1+cu128",
+    "reason": "requirements.compactifai.lock.txt was exported from an x86_64 box. torch 2.4.1+cu121 ships no aarch64 wheel, and the whole nvidia-*-cu12==12.1.x stack it depends on is x86_64/win_amd64 only. Independently, CUDA 12.1 predates Blackwell: GB200 is sm_100 and needs CUDA 12.8+. Every other package in the lock is installed at the locked version. This deviation is FORCED, not chosen, and applies to the 13B numbers but NOT to the already-recorded 7B numbers.",
+    "implication": "13B results in this protocol are not same-stack comparable to the 7B results in eval_protocol.json. To claim a controlled 7B-vs-13B comparison, re-run the 7B under this stack."
+  },
+  "added_packages": {
+    "sentencepiece": "0.2.2",
+    "protobuf": "7.36.1",
+    "reason": "ADDITIVE, not a version change: neither package appears in requirements.compactifai.lock.txt at any version. The 7B repo ships a prebuilt fast tokenizer (tokenizer.json), so transformers never needed to convert anything; the 13B repo ships only tokenizer.model (the raw sentencepiece model), and converting it requires both libraries. Without them AutoTokenizer.from_pretrained raises ImportError before any compute starts.",
+    "equivalence_check": "The converted 13B tokenizer was compared against the 7B's prebuilt tokenizer.json on four probes (English prose, Python source, an MMLU-shaped few-shot prompt, and Japanese text): identical input_ids on all four, identical vocab_size 32000, identical bos/eos/pad ids 1/2/0. Both repos carry the same LLaMA-2 sentencepiece model, so the conversion reproduces the tokenizer rather than approximating it."
+  },
+  "assumptions": [
+    "Model source: mirrors the 7B decision. We use the non-gated NousResearch/Llama-2-13b-chat-hf pinned at commit d73f5fa9c4bc135502e04c27b39660747172d76b, because meta-llama/Llama-2-13b-chat-hf is manually gated. All three safetensors shards match the gated repo's byte-for-byte in SIZE (9,948,693,272 / 9,904,129,368 / 6,178,962,272); shard sizes agreeing is consistent with a straight redistribution but is NOT a content hash, so this is evidence, not proof. Reproduction level for weights: close (unofficial host, contents not independently hashed against Meta's).",
+    "Every protocol field except model_id/model_revision/hardware is copied UNCHANGED from eval_protocol.json. Shot counts, metric keys, dtype, seed, batch_size and score_scale are identical, so the 7B and 13B rows differ only in the model -- which is the whole point of the scaling claim. Do not tune any of these per-scale.",
+    "Architecture: hidden 5120, intermediate 13824, 40 layers, vocab 32000 -> 13,015,864,320 dense parameters. These are read from the model config at runtime by scripts/whitened_lowrank.py:_arch_dims(), not hardcoded here, so this file cannot drift from the actual weights.",
+    "Compression budget is set to match the 7B's ACHIEVED ratio, not its absolute size: 4,023,119,424 / 6,738,415,616 = 0.597042 (40.30% reduction). Applied to 13B that is 7,771,022,084 total, of which 634,388,480 is the 5% sparse residual, giving --target-params-b 7.1366 for the ring cores + embeddings + norms.",
+    "TriviaQA remains the least certain task, exactly as documented in the 7B protocol. The same 5-shot exact_match (remove_whitespace) convention is locked here so the 7B-vs-13B delta stays meaningful even where the absolute deviates from the paper."
+  ]
+}