PhiloLM
PhiloLM is an open language model with 61 million parameters, trained from scratch on philosophy and psychology.
Try it in your browser, where it runs on your CPU. Its training data is PhiloText.
Highlights
- Fast on a CPU. The browser engine loads 98 MB of 8-bit weights and writes about 250 tokens a second on an Intel Core i5-14400F with four threads.
- Open. Weights and code are under Apache-2.0. PhiloText lists every training document with its source, URL and licence. It gives the text for 87% of the training tokens and links for the rest (the encyclopedias and reviews, which are all rights reserved).
- Philosophy and psychology. Four fifths of the 451 million training tokens are web pages, encyclopedias, books, reviews, textbooks and articles in these fields. The rest is general educational web pages. On reference works and books, it beats Gemma 3 270M, which has three times its parameters and 4,000 times its training tokens.
- Architecture. A hashed bigram embedding from the modded-nanogpt speedrun adds 31M parameters looked up without matrix multiplication. Qwen3.8-Flash-Next uses a related n-gram embedding at scale. Three local attention layers alternate with one global layer that uses grouped-query attention.
- Training. Muon trains the hidden matrices. The context grows from 1,024 to 2,048 tokens, the last pass keeps the encyclopedias, reviews, Wikipedia and textbooks, and the released weights are blended with their moving average.
Quick start
pip install torch safetensors tokenizers huggingface_hub
hf download gustavecortal/PhiloLM --local-dir PhiloLM
cd PhiloLM
python generate.py "The problem of personal identity"
generate.py samples as the web page does: presence penalty 2.0, then temperature 0.7, top-k 20 and top-p 0.8. The prompt follows an end-of-text token, so it starts a new document, as in training.
The problem of personal identity is the problem of how we can be both identical to ourselves and different from
other selves.
The model does not load with transformers. In Python, from the downloaded folder:
import json
import torch
from safetensors.torch import load_file
from tokenizers import Tokenizer
from model import Transformer
config = json.load(open("config.json"))
model = Transformer(config["model"], config["vocab_size"])
model.load_state_dict(load_file("model.safetensors"))
model.eval()
tokenizer = Tokenizer.from_file("tokenizer.json")
ids = [config["eos_id"]] + tokenizer.encode("According to Kant,", add_special_tokens=False).ids
with torch.no_grad():
logits = model(torch.tensor([ids]), last_only=True) # next-token logits, shape (1, 1, 32768)
model.py holds the architecture and decode.py the cached decoding, which reuses each layer's keys and values. They are a plain PyTorch reference: the fast path is the browser engine (see In the browser).
Intended uses
- Continuing philosophical and psychological text, on a CPU or in a browser.
- Research on small language models trained on a single domain, and a base for fine-tuning.
It is out of scope for medical, psychological or other advice.
Model overview
- Type: causal language model (base, not instruction-tuned)
- Training stage: pre-training
- Number of parameters: 80M (61M non-embedding), plus 31M n-gram embedding
- Hidden dimension: 576
- Vocabulary: 32,768 tokens, the embedding tied with the output layer
- N-gram embedding: 245,760 rows of 128 values, indexed by a hash of each token and the one before it. A second hash picks a fixed random Β±1 sign pattern, so bigrams that share a row separate. It is added to the first 128 hidden dimensions before every block, with a learned weight per block
- Number of layers: 16
- Hidden layout: 4 Γ (3 Γ (local attention β SwiGLU) β 1 Γ (global attention β SwiGLU))
- Local attention: 9 heads of 64 dimensions, a window of 256 tokens
- Global attention: 9 query heads and 3 key-value heads of 64 dimensions. Half of each key's dimensions carry rotary embedding, and the other half are taken from the previous token's key
- Rotary position embedding dimension: 32 of 64 (base 10,000)
- Attention: QK-norm and a sigmoid gate on each head's output
- SwiGLU intermediate dimension: 1,512
- Context length: trained on 2,048-token windows, evaluated at 4,096. Documents are separated by the end-of-text token, and each attends only to itself
- Tokenizer: byte-level BPE trained on the training text
Training
The training text holds 450.8 million tokens. Validation and test splits (8.6 million tokens each) are held out by work, so no book, article or entry appears in two splits. PhiloText lists every document.
| Source | Share of training tokens |
|---|---|
| Essential-Web pages on philosophy and psychology | 26.8% |
| FineWeb-Edu general educational pages | 19.0% |
| Project Gutenberg, Standard Ebooks and Wikisource books | 14.3% |
| FineWeb-Edu pages on philosophy and psychology | 12.0% |
| Wikipedia (WikiProjects Philosophy and Psychology) | 10.2% |
| Stanford Encyclopedia of Philosophy | 6.5% |
| Notre Dame Philosophical Reviews | 4.2% |
| PubMed Central review articles (CC BY) | 3.0% |
| Internet Encyclopedia of Philosophy | 2.3% |
| Classics in the History of Psychology, open textbooks, Noba | 1.9% |
The model trained on 1.45 billion tokens in four passes over the data, reading no document more than four times. The third pass leaves out FineWeb-Edu. The fourth keeps the encyclopedias, reviews, Wikipedia, textbooks and half of the primary philosophical works. Windows were 1,024 tokens long for the first 25% of training, then 2,048. Hidden matrices used Muon (learning rate 0.005) and the rest AdamW (0.003), with a cosine schedule down to 10%. The released weights blend the final weights equally with an exponential moving average over the last 17% of training.
It trained in 15.2 hours (88,777 updates) on one Intel Arc A770 16 GB.
Evaluation
On the test split (3,048 documents, 40.8 MB), PhiloLM reaches 0.9368 bits per byte in 4,096-token windows.
Bits per byte does not depend on the tokenizer, so recent small base models can be scored on the same documents. Each document is scored alone, in 2,048-token windows, after the model's beginning-of-text token and in its own tokenizer (lower is better). Web pages are Essential-Web and FineWeb-Edu (20.7 MB); reference works are the encyclopedias, reviews, Wikipedia, textbooks and PubMed Central (12.1 MB); books are Project Gutenberg, Standard Ebooks and Classics in the History of Psychology (8.0 MB).
| Model | Parameters, without hashed tables | Non-embedding parameters | Training tokens | All | Web pages | Reference works | Books |
|---|---|---|---|---|---|---|---|
| PhiloLM | 80.2M | 61.4M | 1.45B | 0.944 | 0.947 | 0.884 | 1.029 |
| SmolLM2-135M | 134.5M | 106.2M | 2T | 0.840 | 0.797 | 0.836 | 0.957 |
| LFM2.5-230M-Base | 229.7M | 162.6M | 28T | 1.048 | 0.949 | 1.035 | 1.321 |
| Gemma 3 270M | 268.1M | 100.3M | 6T | 0.940 | 0.870 | 0.935 | 1.130 |
| SmolLM2-360M | 361.8M | 314.6M | 4T | 0.773 | 0.731 | 0.768 | 0.891 |
| Qwen3-0.6B-Base | 596.0M | 440.5M | 36T | 0.793 | 0.745 | 0.786 | 0.930 |
PhiloLM's 31.5M-parameter bigram table is left out of its count, since it is looked up rather than multiplied. Parameters are counted from each model's weights, and training tokens come from each model card (Qwen3's is the family's figure). PhiloLM beats LFM2.5-230M on every kind of text. On reference works and books, it also beats Gemma 3 270M, which has three times its parameters and 4,000 times its training tokens. SmolLM2 and Qwen3 do better, above all on web pages. These models were trained on web corpora that may contain some of the test documents, which would favour them.
The browser engine's quantization raises bits per byte by 0.2%, measured on 60 validation documents (their first 1,024 tokens): 0.9349 in float32 against 0.9369 in core.bin.
In the browser
core.bin (98 MB) and core.wasm are the form the web page runs, in an engine written in Rust and compiled to WebAssembly. Matrices are 8-bit integers with one scale per row, each matrix's input is quantized to 8 bits per token, and the bigram table is 4-bit. SmoothQuant (alpha 0.7) moves activation outliers into the weights before quantization. Keys and values are cached in bfloat16.
The engine reads a prompt 16 tokens at a time. Up to three Web Workers share each matrix product and each layer's attention. On an Intel Core i5-14400F the page writes about 250 tokens a second with four threads. core-plain.wasm serves browsers without WebAssembly SIMD.
Limitations
- It is a base model: it continues text and does not follow instructions or answer questions.
- It invents facts, dates and attributions with the confidence of an encyclopedia. Asked about Freud's theory of dreams, it wrote that the theory "was first introduced in 1890 by Wilhelm Roux".
- It sometimes repeats itself and loses an argument's thread over long continuations.
- English only. Its sources include 19th and early 20th century books, with their period's views.
Licence
Weights and code: Apache-2.0. The training documents keep their own licences, listed for each document in PhiloText.
Citation
@misc{cortal2026philolm,
author = {Cortal, Gustave},
title = {PhiloLM: a small language model trained on philosophy and psychology},
year = {2026},
url = {https://huggingface.co/gustavecortal/PhiloLM}
}
- Downloads last month
- -