Add benchmark methodology disclosure
Browse files
README.md
CHANGED
|
@@ -87,3 +87,7 @@ PicoLM-V2.1 runs out of the box on mobile devices via PocketPal AI and MobAI:
|
|
| 87 |
- **File:** `picolm-v2.1-81m-instruct-fp16.gguf`
|
| 88 |
- **Memory Footprint:** ~175 MB RAM
|
| 89 |
- **Mobile Throughput:** ~40-45 tokens/sec
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
- **File:** `picolm-v2.1-81m-instruct-fp16.gguf`
|
| 88 |
- **Memory Footprint:** ~175 MB RAM
|
| 89 |
- **Mobile Throughput:** ~40-45 tokens/sec
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
> **Benchmark Methodology Note:** The 42.00% ARC-Easy score reported for PicoLM-V2 / V2.1 was evaluated on a legacy, non-standardized 250-question unnormalized sample. Standardized, full-suite evaluations under the official EleutherAI LM-Evaluation-Harness (character-normalized) were introduced starting with **PicoLM-V3**.
|