Add CHERRY-1.8B-CYBER (TeamSparta, self-reported): CyberMetric-80 = 75.00% + CyberBench Average = 26.6
Add CHERRY-1.8B-CYBER (TeamSparta) — self-reported results
This PR adds self-reported, held-out evaluation results for CHERRY-1.8B-CYBER, a 1.8B-parameter cybersecurity-specialized SLM by TeamSparta (open weights: https://huggingface.co/teamsparta-inc/k-ax-spartan-cherry-1.8b).
Results added
- CyberMetric: 75.00% (60/80) on CyberMetric-80, added under the existing
80 Q (Accuracy)metric. - SecBench (new benchmark entry): 69.67% (209/300) on a reproducible subset — SecBench
MCQs_2730filtered to canonical single-answer 4-option MCQs, fixed seed-42 sample of 300 (MCQ 300 Q subset (Accuracy)).
Disclosure (honest self-report)
- The model is domain-specialized for cybersecurity (not a general-purpose frontier model); the model name is marked accordingly, incl. parameter count (1.8B).
- Scores are self-reported: single-letter MCQ harness, greedy decoding (do_sample=False), system prompt included in the eval JSON. Benchmark items are held out from training data.
- Reproduction: per-question outputs with prompts and decoding config: https://huggingface.co/teamsparta-inc/k-ax-spartan-cherry-1.8b/tree/main/eval (
cm80_cyber.json,sb300_cyber.json). - No existing entries were modified — strictly additive (verified by deep-compare against the current
results.json).
Happy to adjust format or provide any additional detail. Thanks for maintaining the Observatory!
Cleaned up the diff: this PR now adds exactly ONE line — CHERRY-1.8B-CYBER = 75.00 under CyberMetric "80 Q (Accuracy)". Original CRLF line endings preserved; no other benchmark or value is touched. (Reverted an earlier commit that re-serialized the whole file and added a solo SecBench entry.) Score is self-reported, held-out, reproducible: per-question JSON at https://huggingface.co/teamsparta-inc/k-ax-spartan-cherry-1.8b/tree/main/eval
Added CyberBench results for the same model row (CHERRY-1.8B-CYBER, self-reported): Average = 26.6, plus all 10 sub-task metrics under the existing CyberBench columns.
Sub-task scores: CyNER F1 2.7, APTNER F1 1.6, CyNews R-1/2/L 9.5/2.2/7.9, SecMMLU 51.0, CyQuiz 60.0, MITRE 11.0, CVE 3.5, Web F1 44.9, Email F1 82.3, HTTP F1 2.5. The profile is consistent with the model's MCQ-focused specialization: multiple-choice is above the similar-size board baselines (e.g. Falcon-7B SecMMLU 27.0 / CyQuiz 27.0), while structured NER/JSON extraction and strict exact-match classification are weak. Reported as-is, no protocol shopping.
Protocol — replica of the official harness (jpmorganchase/CyberBench) defaults: alpaca prompt as raw completion (no chat template — same as the existing rows, incl. the chat models), 5-shot with all-mpnet-base-v2 semantic example selection from the train split (0-shot for CyNews), greedy decoding, newline stop, official metric code (strict exact-match accuracy for SecMMLU/CyQuiz/MITRE/CVE; binary F1 with positives web=phishing, email=phishing, http=anomalous; corpus-level micro-F1 over JSON entity sets for CyNER/APTNER; ROUGE-1/2/L aggregate for CyNews). Average = mean of the 10 task scores with CyNews as mean(R-1/2/L) — this formula reproduces the published board rows (e.g. GPT-3.5-Turbo 62.61 vs 62.6, Falcon-7B 39.36 vs 39.4).
Disclosed deviations (bounded compute, fixed a priori): evaluated on fixed seed-42 test subsets — SecMMLU and CyQuiz on their full test sets (100 each); 200 items/task for CyNER, APTNER, MITRE, CVE, Web, Email, HTTP; 150 for CyNews (official runs use the full test splits). Batched greedy generation (batch 8, left padding). Per-item outputs (prompt, gold, response) for every task: https://huggingface.co/teamsparta-inc/k-ax-spartan-cherry-1.8b/tree/main/eval/cyberbench_v1
Contamination audit: the model's cybersecurity SFT data was string-matched (normalized substring, CJK-aware) against every sampled CyberBench test item: 0 matches across all 10 tasks (1750 sampled test items, normalized-substring vs the full cyber SFT corpus; items under 40 normalized chars — mostly short Web URLs — cannot be audited by substring but their datasets were never training sources). CyberBench was never part of the training data.
Happy to re-run any task on the full test split if useful.