aethertp commited on
Commit
d76ff5f
·
verified ·
1 Parent(s): 3ffd8fb

Add benchmark methodology disclosure

Browse files
Files changed (1) hide show
  1. README.md +4 -0
README.md CHANGED
@@ -87,3 +87,7 @@ PicoLM-V2.1 runs out of the box on mobile devices via PocketPal AI and MobAI:
87
  - **File:** `picolm-v2.1-81m-instruct-fp16.gguf`
88
  - **Memory Footprint:** ~175 MB RAM
89
  - **Mobile Throughput:** ~40-45 tokens/sec
 
 
 
 
 
87
  - **File:** `picolm-v2.1-81m-instruct-fp16.gguf`
88
  - **Memory Footprint:** ~175 MB RAM
89
  - **Mobile Throughput:** ~40-45 tokens/sec
90
+
91
+
92
+
93
+ > **Benchmark Methodology Note:** The 42.00% ARC-Easy score reported for PicoLM-V2 / V2.1 was evaluated on a legacy, non-standardized 250-question unnormalized sample. Standardized, full-suite evaluations under the official EleutherAI LM-Evaluation-Harness (character-normalized) were introduced starting with **PicoLM-V3**.