Update CyberSLM-33M-Base / -Instruct with retrained checkpoints
Update CyberSLM-33M-Base / -Instruct (retrained)
This updates the two existing CyberSLM rows rather than adding new ones: the
models have been retrained from scratch after fixing 19 defects in the
training pipeline, and these rows now point at the retrained checkpoints.
Pretrained from scratch on a 194.8M-token corpus that is ~60%
cybersecurity, ~20% general English, ~15% programming, ~5% CS fundamentals.
Open weights, training checkpoints, not merged.
- Base: https://huggingface.co/sabari2005/cyberslm-base
- Instruct: https://huggingface.co/sabari2005/cyberslm-instruct
- Code: https://github.com/Sabari2005/cyberslm
Setup
| harness | lm-evaluation-harness 0.4.9 (HellaSwag, ARC-E, ARC-C, PIQA) |
| arithmark | AxiomicLabs' bencharithmark-3.py and benchmark_arithmark-2.0.py, unmodified |
| n-shot | 0 |
| dtype | float32 |
| stack | torch 2.13.0+cu130, transformers 4.57.6, datasets 5.0.1 |
| hardware | NVIDIA A10G |
| architecture | 12L, d_model 384, 6 heads, SwiGLU 1024, RoPE, RMSNorm, tied head, ctx 2048, vocab 32,000 |
Results
Both acc and acc_norm are given, because the column convention is not
documented. The headline row uses the mapping I inferred by calibrating
against the SmolLM2-135M row already on the board (piqa tracks acc;hellaswag and arcChall track acc_norm; arc tracks arc_easy acc). If that
mapping is wrong, please use whichever column is correct from the table below.
| model | metric | ARC-E | HellaSwag | PIQA | ARC-C | ArithMark-3 | ArithMark-2 |
|---|---|---|---|---|---|---|---|
| CyberSLM-base | acc | 32.79 | 26.35 | 54.68 | 18.17 | 31.00 | 26.96 |
| CyberSLM-base | acc_norm | 32.24 | 26.84 | 52.23 | 22.70 | 31.00 | - |
| CyberSLM-instruct | acc | 32.37 | 26.51 | 53.81 | 19.11 | 30.00 | 26.88 |
| CyberSLM-instruct | acc_norm | 32.15 | 26.19 | 52.88 | 22.35 | 30.20 | - |
Honest note on these numbers
These scores sit at or near chance on HellaSwag, PIQA and ARC-Challenge, and
are comparable to models an order of magnitude smaller. That is expected and I
would rather state it than have it discovered: the corpus is domain-specific,
and these benchmarks measure everyday commonsense, physical reasoning and
grade-school science, none of which the training data covers.
The pipeline itself is verified, so the numbers are not an artefact of a broken
setup: the HF export reproduces the original implementation's logits to 3.1e-05
with identical greedy decoding, and the tokenizer matches SentencePiece exactly
on all 9,000 ArithMark-3 strings. Everything is reproducible from the repo.
Happy to re-run under different settings if you use a different n-shot or
metric convention.
Hi Sabari, cool models, and thank you for your submission, however all benchmarks should be acc norm!