| --- |
| license: mit |
| language: |
| - en |
| library_name: ghostlm |
| pipeline_tag: text-generation |
| tags: |
| - ghostlm |
| - from-scratch |
| - generalist |
| - cybersecurity |
| - small-language-model |
| --- |
| |
| # ghost-small-gen |
|
|
| A small (~45M parameter) decoder-only language model **trained entirely from |
| scratch** in PyTorch, no pretrained weights and no fine-tuning. It is the first |
| generalist checkpoint in the [GhostLM](https://github.com/joemunene-by/GhostLM) |
| project: a model that broadened from a cybersecurity-only corpus into a small |
| generalist while keeping cybersecurity as its deepest specialty. |
|
|
| - Parameters: ~45M (6 layers, d_model 512, RoPE + SwiGLU + RMSNorm) |
| - Tokenizer: GPT-2 BPE (50,257) + 7 special tokens |
| - Training: 30,000 steps on a Mac M4 (MPS), final val_loss 3.76 |
| - Corpus: a decontaminated 258.9M-token multi-domain corpus that is only |
| **8.6% cybersecurity** (general web, broad Wikipedia, code, math, instruction, |
| and a cybersecurity layer), with 0.004% measured benchmark contamination |
| - Recipe: intra-document attention masking + a multi-stage domain curriculum |
|
|
| ## Evaluation (honest, full benchmark sets) |
|
|
| Debiased multi-permutation text-scoring. `+` means the 95% bootstrap CI lower |
| bound is above the 25% random baseline (significantly above chance). Peer |
| numbers are published zero-shot references for the small-model class; harnesses |
| differ, so treat them as context, not an exact comparison. |
|
|
| | Benchmark | n | ghost-small-gen (45M) | 95% CI | vs random | Peer reference | |
| |---|---:|---:|---:|:--:|---| |
| | ARC-Easy | 2365 | 27.2% | 25.4-28.9 | + | Pythia-160M 43.5, 111M 34.8, 256M 37.6 | |
| | ARC-Challenge | 1165 | 24.3% | 22.1-26.6 | ~ | Pythia-160M 18.8, SmolLM2-360M 36.6 | |
| | OpenBookQA | 500 | 27.4% | 23.7-31.1 | ~ | 111M 27.8, 256M 25.4, LaMini-35M 26.2 | |
| | SecQA (cyber) | 210 | 34.3% | 28.5-40.6 | + | retained specialty | |
| | CTF eval (cyber) | 30 | 63.3% | 46.7-80.0 | + | retained specialty | |
|
|
| What this says, calibrated for a 45M model trained from scratch on free compute: |
|
|
| - The generalist pivot worked. Three of five benchmarks are statistically above |
| the 25% random baseline, on a corpus only 8.6% cybersecurity. |
| - Cybersecurity is fully retained and is the standout (SecQA 34.3%, CTF 63.3%). |
| - It is competitive within its own size class: OpenBookQA beats the 256M and 35M |
| survey peers, and ARC-Challenge beats Pythia-160M, a roughly 3.5x larger model. |
| - It does not clear the 35-45% competitive band on ARC-Easy (27.2%): above |
| chance, not state of the art there. |
|
|
| This is a solid, defensible result for the size and the compute, not a |
| "beats everything" claim. |
|
|
| ## Intended use and limitations |
|
|
| Research and education: a transparent, hand-written small model for studying |
| from-scratch training, generalist corpus design, and cybersecurity-aware |
| language modeling. It is a base model (not instruction-tuned), small, and will |
| hallucinate. Do not use it for safety-critical decisions. The cybersecurity |
| content is for defensive and educational understanding. |
|
|
| ## How to load |
|
|
| This is a custom architecture, not a `transformers` model. Load it with the |
| GhostLM code: |
|
|
| ```python |
| import torch |
| from safetensors.torch import load_model |
| from ghostlm.config import GhostLMConfig |
| from ghostlm.model import GhostLM |
| from huggingface_hub import hf_hub_download |
| import json |
| |
| repo = "Ghostgim/ghost-small-gen" |
| cfg = GhostLMConfig(**{k: v for k, v in json.load(open(hf_hub_download(repo, "config.json"))).items() |
| if k in GhostLMConfig().__dataclass_fields__}) |
| model = GhostLM(cfg) |
| load_model(model, hf_hub_download(repo, "model.safetensors")) |
| model.eval() |
| ``` |
|
|
| See the [GhostLM repository](https://github.com/joemunene-by/GhostLM) for the |
| model code, tokenizer, generation utilities, and the full scorecard. |
|
|
| ## Training details |
|
|
| - Base architecture: ghost-small-v0.5 (RoPE, SwiGLU, RMSNorm) |
| - Context length: 512, batch 16, grad-accum 4, lr 3e-4, 2000-step warmup |
| - Intra-document attention masking so packed documents do not attend across |
| EOS boundaries |
| - Multi-stage domain curriculum that shifts the data mixture across training |
| (broad web early, code/math/knowledge upweighted later) |
| - Corpus decontaminated against every evaluation benchmark before training |
|
|
| ## License |
|
|
| MIT. Built and trained by Joe Munene. |
|
|
