Upload tr3/heal/llama2-7b-base/eval_protocol_llama2_7b_base.json with huggingface_hub

tr3/heal/llama2-7b-base/eval_protocol_llama2_7b_base.json

@@ -0,0 +1,48 @@
+{
+  "model_id": "NousResearch/Llama-2-7b-hf",
+  "model_revision": "8efe6c9b93655b934e27bd9981e3ec13e55aee9d",
+  "lm_eval_revision": "0.4.12",
+  "tasks": {
+    "mmlu": "mmlu",
+    "hellaswag": "hellaswag",
+    "boolq": "boolq",
+    "triviaqa": "triviaqa",
+    "gsm8k": "gsm8k"
+  },
+  "num_fewshot": {
+    "mmlu": 5,
+    "hellaswag": 10,
+    "boolq": 0,
+    "triviaqa": 5,
+    "gsm8k": 5
+  },
+  "apply_chat_template": false,
+  "batch_size": "auto",
+  "dtype": "float16",
+  "seed": 1234,
+  "score_scale": "fraction",
+  "result_parser": {
+    "mmlu": "acc,none",
+    "hellaswag": "acc_norm,none",
+    "boolq": "acc,none",
+    "triviaqa": "exact_match,remove_whitespace",
+    "gsm8k": "exact_match,strict-match"
+  },
+  "hardware": {
+    "gpus_max": 4,
+    "gpu_model": "NVIDIA RTX A6000",
+    "note": "Shared box; this reproduction is capped at 4 free GPUs. Paper hardware was 8x A10g and is NOT matched."
+  },
+  "assumptions": [
+    "Model source: the paper uses gated meta-llama/Llama-2-7b-chat-hf. This box has no HF access token, so we use the non-gated community mirror NousResearch/Llama-2-7b-chat-hf, pinned at commit 351844e75ed0bcbbe3f10671b3c808d2b83894ee. The mirror redistributes Meta's original weights bit-for-bit; provenance is recorded here and in assumptions.md. Reproduction level for weights: close (identical weights, unofficial host).",
+    "lm-evaluation-harness revision: the paper does not disclose it. We pin the pip release lm_eval==0.4.12 (new subcommand CLI). Task ids: mmlu, hellaswag, boolq, triviaqa, gsm8k, all present in this release.",
+    "num_fewshot: the paper does not disclose per-task shot counts. We adopt Open-LLM-Leaderboard-v1-style conventions where they exist: MMLU 5-shot, HellaSwag 10-shot, BoolQ 0-shot, GSM8K 5-shot. These three (MMLU/BoolQ/GSM8K) reproduce the paper's Original-row numbers closely (46.41 / 79.76 / 23.05), which is our evidence the paper used an lm-eval-style harness at these shot counts.",
+    "TriviaQA is the least certain: the paper reports 19.03 for the chat model, which is unusually low for few-shot TriviaQA. We lock 5-shot exact_match (rc.nocontext, remove_whitespace filter, ignore_case/ignore_punctuation) as a defensible standard and EXPECT an absolute deviation from the paper here. Because all five model variants share this identical protocol, the scientific claim (relative degradation under compression) is preserved even where the TriviaQA absolute differs.",
+    "HellaSwag shot count (10) and metric (acc_norm) follow leaderboard convention; small absolute gap vs paper 80.55 is expected and reported as deviation, not corrected by tuning the protocol.",
+    "apply_chat_template=false: although Llama-2-7b-chat is instruction-tuned, the paper's MMLU 46.41 matches the standard no-template 5-shot number for this model, indicating the paper evaluated without the chat template. We therefore evaluate all tasks with no chat template for internal consistency.",
+    "dtype=float16 for evaluation. The paper's Original row is a float32 checkpoint (27.1GB); fp16 eval is standard and the accuracy delta from precision is negligible. This is documented rather than matched to float32.",
+    "batch_size=auto, seed=1234 fixed across all variants and all stages.",
+    "score_scale=fraction: every metrics JSON in this project stores lm-eval native fractions in [0,1]; the verifier multiplies by 100 to compare against paper percentages. This must not be changed per-stage.",
+    "Result parser keys are the exact lm-eval 0.4.12 metric keys: MMLU acc, HellaSwag acc_norm, BoolQ acc, TriviaQA exact_match (remove_whitespace filter), GSM8K exact_match (strict-match filter). These are locked now to avoid a silent metric-key change after baseline_smoke."
+  ]
+}