PicoLM-V4-85M

PicoLM-V4-85M is a base language model with 84,960,832 parameters, trained from scratch for 6.0 billion training tokens (part of the data was seen more than once, see below) on English web text, textbook-style synthetic text, mathematics, Wikipedia and Python code. It is the fourth model in the PicoLM series. It has not been instruction-tuned and does not follow instructions reliably; it continues text.

The weights are stored in the Qwen3 format of the transformers library (Qwen3ForCausalLM), so the model loads with AutoModelForCausalLM and AutoTokenizer without trust_remote_code. This was tested on transformers 4.57.1 and 5.22.1. The tokenizer is the 24,576-token BPE vocabulary of PicoLM-V3.

This card reports only numbers that were measured with one evaluation setup, described below (one exception is noted under the table). Where a number could not be measured, the card says so.

Intended use

The model is meant for research on small language models, for use as a starting point for fine-tuning, and for experiments that need a model small enough to run on a CPU. It is not suitable for question answering, summarization or any setting where factual accuracy matters. At this size the model produces fluent text that is frequently wrong, and it repeats itself in longer generations.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "aethertp/PicoLM-V4-85M"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float32)

prompt = "The capital of France is"
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=40, do_sample=False, repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))

Output of the code above, identical on transformers 4.57.1 and 5.22.1 (CPU, fp32). It shows the repetition and the factual drift mentioned under Limitations:

The capital of France is Paris, the capital of the United Kingdom.
The capital of the United Kingdom is London, the capital of the United Kingdom.
The capital of the United Kingdom is London, the capital of the

The tokenizer does not add a beginning-of-text token. During training every document starts after an <|endoftext|> separator, so you can also prepend tok.bos_token to the prompt. The second row of the evaluation table below does this. In that table it changes most scores by about one standard error or less; the largest shift is LAMBADA perplexity (48.4 to 45.2), and eval_results.json shows a larger shift for SciQ acc_norm (+2.9 points). Both ways of prompting work.

Model details

Parameters	84,960,832 (input and output embeddings are tied and counted once)
Layers	20
Hidden size	576
Attention	9 query heads, 3 key/value heads, head dimension 64
Feed-forward	SwiGLU, intermediate size 1536
Normalization	RMSNorm (epsilon 1e-5), plus RMSNorm on queries and keys
Positions	RoPE, base 10,000
Vocabulary	24,576
Context length	2048
Initialization	normal, standard deviation 0.02

The tokenizer was inherited from PicoLM-V3 and was not retrained. Token id 0 is <|endoftext|>, which separates documents during training. Token id 2 is <|im_end|>; it is not used as a separator in the pretraining data, and a chat fine-tune can use it to mark the end of a turn. The end-of-sequence token set in this repository is id 0, <|endoftext|>, and id 3 is <|pad|>.

PicoLM-V3 used a different architecture (21 blocks applied twice, a factorized embedding of rank 128, 82.2M parameters) and was trained on about 757 million tokens. V4 shares only its tokenizer with V3.

Training data

Training had two stages. Stage 1 used 80% of the optimizer steps (36,620 steps, sequence length 1024). Stage 2 used the remaining steps (9,156 steps, sequence length 2048) with a different mix that gives more weight to higher-quality filtered text and includes the learning-rate decay.

Stage 1 (pool of 6,000,007,358 tokens)

Source	Tokens	Share
FineWeb-Edu (deduplicated, from SmolLM-Corpus)	2,700,002,868	45%
DCLM-Baseline	1,500,000,908	25%
Cosmopedia v2	900,000,338	15%
Python-Edu	300,002,011	5%
FineMath-3+	300,000,861	5%
Wikipedia (English, 2023-11-01)	300,000,372	5%

Stage 2 (pool of 1,200,004,547 tokens)

Source	Tokens	Share
FineWeb-Edu, educational score 4 or higher	540,000,272	45%
Cosmopedia v2	240,000,457	20%
DCLM-Baseline	180,000,146	15%
FineMath-4+	120,002,083	10%
Python-Edu (non-English-comment files removed, see below)	60,001,053	5%
Wikipedia	60,000,536	5%

Documents were tokenized separately, joined with <|endoftext|>, and written to token files. During training, windows of 1024 or 2048 tokens were cut at random positions from these files, with the files chosen in proportion to their size. Windows can span document boundaries, and attention is not masked at those boundaries.

Properties of the data that affect how to read the results

The mix was not tuned. The proportions come from published recipes for small models. No ablation was run at this scale, so they may not be the best choice for an 85M model.

Part of the data was seen more than once. Windows were sampled with replacement, and the number of sampled positions is not far below the size of the pool. In stage 1, 4.80 billion positions were drawn from a 6.0 billion token pool. Under a random-window model, about 45% of the pool was never drawn, about 36% was drawn once and about 19% was drawn two or more times. About 31% of the tokens processed in stage 1 were repeat views of text the model had already seen. A simulation of the sampler agrees with these figures. Stage 2 draws 1.2 billion positions from a 1.2 billion token pool, so the corresponding figures are about 37% never drawn, 37% once and 26% two or more times, and about 37% of the tokens processed were repeats. The 6.0 billion tokens reported for the whole run are therefore tokens processed, and the number of distinct tokens seen is lower. These figures are expected values from the sampling scheme, not counts from the run.

The Python subset contains non-English comments. A heuristic check of comments and docstrings (stopword lists for Spanish, Portuguese, French, German, Italian, Indonesian, Turkish and Polish, plus the share of non-ASCII characters) was run on a sample of the Stage 1 Python data: 6 slices of 3 million tokens, 32,271 files. It flagged 9.8% of the files as having mostly non-English comments. Comments and docstrings make up 29.3% of the characters in those files, and non-English comment text is 2.35% of all Python characters. Python is 5% of the Stage 1 mix, so this is roughly 0.1% of all Stage 1 training characters. The comments were mostly Chinese, followed by Portuguese, Spanish, Korean, Russian, Polish and French. The check does not look at string literals or identifiers. Stage 1 Python was not filtered. For Stage 2, files flagged by the same heuristic were removed: 12,337 of 119,808 scanned files (10.3%). The heuristic's false-drop rate on English standard-library code was 0.03%.

Stage 2 documents are almost entirely different from Stage 1 documents. Stage 2 data was prepared in separate runs, using slices of the source datasets chosen to avoid the Stage 1 slices, and the result was checked afterwards by looking for exact duplicates: a Stage 2 document counts as a duplicate if a Stage 1 document from the same source has the same length, the same first 128 tokens and the same last 32 tokens. By that test, 0.86% of the Stage 2 Cosmopedia documents (0.93% of its tokens), 0.18% of the DCLM documents (0.16% of tokens) and 0.93% of the FineMath-4+ documents (0.89% of tokens, compared against FineMath-3+) also occur in Stage 1. For FineWeb-Edu, Python-Edu and Wikipedia the overlap is 11, 1 and 0 documents. Near-duplicates are not detected by this test, and the check says nothing about duplicates inside either stage.

The validation data comes from the same distribution as the training data. The held-out text was taken from the same sources as the training pool (about 3 million tokens per source) and the model was trained on that mix, so validation loss is more favorable to this model than a neutral set would be. It is used here only to compare this model with PicoLM-V3, which shares the tokenizer, and the comparison inherits this bias.

Training

Optimizer	Muon for the two-dimensional weight matrices of the transformer blocks, AdamW for the embedding and the normalization weights
Learning rate	3e-3 peak, for both optimizers
Weight decay	0.1 on the Muon parameters, none on the AdamW parameters
AdamW	betas (0.9, 0.95), epsilon 1e-8
Muon	momentum 0.95
Schedule	1,000 steps of linear warmup, constant until 80% of the steps, then a 1-sqrt decay to zero over the last 20%
Batch	131,072 tokens per step (8 micro-batches on each of 2 GPUs, 8 sequences of 1024 tokens in stage 1, 4 sequences of 2048 in stage 2)
Steps	45,776 (6.0 billion tokens processed)
Loss	cross-entropy plus a z-loss term of 1e-4 on the log-partition function
Precision	fp16 mixed precision with dynamic loss scaling
Hardware	2 NVIDIA T4 GPUs on Kaggle
Throughput	about 25,400 tokens per second on average over all sessions (per-session averages: 20,000 to 22,000 in the first three sessions, 24,500 to 31,400 in the last four)
Compute	65.6 hours of wall-clock time on 2 T4 GPUs, in 7 sessions

The run was split into sessions because of the platform's time limit. Each session resumed from a checkpoint that contained the model, the optimizer state and the loss-scaler state. The data sampler is seeded from the step number at each resume rather than restored, which only changes which random windows are drawn.

The final training loss and validation loss were 2.458 and 2.2751 nats per token. The training figure is the mean over the last 10 logged steps of the final session (each a single batch of 131,072 tokens, so it is noisy). The validation figure is the unweighted mean over the six held-out files (256 windows of 1024 tokens each, evaluated at step 45776 in fp32). The two numbers are not directly comparable: the training mix puts 45% of its weight on FineWeb-Edu and 15% on DCLM, which are among the held-out files with the highest loss, while the validation mean gives every file the same weight. Weighting the six per-file validation losses by the Stage 2 shares gives 2.54.

Evaluation

All results below were produced with lm-evaluation-harness 0.4.9.1 (transformers 4.57.1, datasets 3.6.0), zero-shot, on the full test or validation sets, with a maximum sequence length of 1024 and fp32 weights. Every model in the table was run with the same code and the same settings. The one exception is described below the table. Perplexity on WikiText is the word-level perplexity reported by the harness; bits per byte is also from the harness. Reference models were evaluated by us under this setup; their numbers may differ from the ones in their own papers or cards.

Model	HellaSwag	ARC-e	ARC-c	PIQA	Winogrande	OBQA	SciQ	BoolQ	LAMBADA acc	LAMBADA ppl	WikiText word ppl	WikiText bits/byte
PicoLM-V4-85M	32.8	54.8	26.8	63.7	50.1	31.4	78.5	59.1	31.2	48.37	38.83	0.9872
PicoLM-V4-85M (with endoftext prefix)	33.3	54.6	26.4	63.4	51.2	31.6	79.7	59.0	31.8	45.16	38.94	0.9880
GPT-2 124M	31.1	43.8	22.7	62.9	51.6	27.2	75.2	48.7	32.6	40.06	37.37	0.9769
Pythia-160M	30.3	43.6	23.6	62.3	51.3	26.8	75.4	56.9	35.4	37.26	36.37	0.9696
OPT-125M	31.3	43.5	22.8	63.0	50.3	28.0	75.2	55.4	37.9	26.02	35.30	0.9615
SmolLM2-135M	43.1	64.4	29.6	68.4	53.0	33.4	84.0	60.3	42.9	19.26	25.53	0.8741
PicoLM-V3 (Instruct)	28.0	41.1	23.2	56.9	50.7	25.0	61.4	38.1	12.4	1349.13	234.38	1.4722
PicoLM-V3 (base)	28.3	42.7	23.5	59.6	52.6	27.8	63.3	61.5	23.0	223.22	83.21	1.1928

Metrics are accuracy in percent unless a column says otherwise: acc_norm for HellaSwag, ARC-Challenge and OpenBookQA, acc for the others. Standard errors and the unnormalized/normalized pairs for every task are in eval_results.json in this repository.

How to read the table. Standard errors from the harness are about 0.5 points for HellaSwag, 1.0 for ARC-Easy, 1.3 for ARC-Challenge, 1.1 for PIQA, 1.4 for Winogrande, 2.1 for OpenBookQA, 1.3 for SciQ, 0.9 for BoolQ and 0.65 for LAMBADA accuracy (LAMBADA perplexity: 1.9). The harness reports no standard error for WikiText. Differences of one to two standard errors are not evidence of a real difference.

• Against GPT-2 124M, Pythia-160M and OPT-125M, PicoLM-V4-85M is clearly ahead on ARC-Easy (54.8 against 43.5 to 43.8) and ahead by about 3 to 4 points on ARC-Challenge (26.8 against 22.7 to 23.6) and by 1.5 to 2.5 points on HellaSwag (32.8 against 30.3 to 31.3). It is ahead by 3 to 5 points on OpenBookQA and by 3 points on SciQ, but those gaps are within about two standard errors. It is level with them on PIQA and Winogrande.
• It is behind them on LAMBADA and WikiText. LAMBADA perplexity is 48.4 against 40.1 (GPT-2), 37.3 (Pythia-160M) and 26.0 (OPT-125M); LAMBADA accuracy is 31.2 against 32.6, 35.4 and 37.9. WikiText word perplexity is 38.8 against 37.4 (GPT-2), 36.4 (Pythia-160M) and 35.3 (OPT-125M). The training mix contains no book corpus; we have not tested whether this explains the gap.
• BoolQ. PicoLM-V4-85M scores 59.1, below the 62.2% that a model reaches by always answering "yes" (the majority class of the validation set). The three small reference models score lower (GPT-2 48.7, Pythia-160M 56.9, OPT-125M 55.4), the gap to Pythia-160M (2.2 points) is less than two standard errors of the difference, while the gaps to OPT-125M and GPT-2 are larger. Scores on this task near or below the majority rate, which includes SmolLM2-135M at 60.3, do not show reading comprehension.
• SmolLM2-135M is better than PicoLM-V4-85M in every column (higher accuracy, lower perplexity and bits per byte). It is clearly ahead on HellaSwag (43.1), ARC (64.4 on ARC-Easy), PIQA, SciQ, LAMBADA (perplexity 19.3) and WikiText; the gaps on Winogrande, OpenBookQA and BoolQ are within about two standard errors. It was trained on about 330 times as many tokens.
• Against PicoLM-V3, PicoLM-V4-85M is better on every column except Winogrande (within error of both V3 rows) and BoolQ, where V3 base scores 61.5 against 59.1, close to the majority rate. For example, WikiText word perplexity is 38.8 against 83.2 (V3 base) and 234.4 (V3 Instruct), and LAMBADA perplexity is 48.4 against 223.2 and 1349.1.

One exception to "same settings": Pythia-160M's BoolQ and WikiText runs ran out of GPU memory in the main evaluation (batch size 32), so those two cells come from a separate run of the same harness, versions and settings with batch size 8; every other cell comes from the main run. BoolQ passages longer than 1024 tokens are truncated from the left for every model, which affects a small number of examples. MMLU and code benchmarks such as HumanEval were not run.

Models used for comparison

Model	Parameters	Training tokens (as published)
GPT-2	124M	not published (WebText)
Pythia-160M	160M	300 billion
OPT-125M	125M	about 180 billion (size of the training corpus; the paper's learning-rate schedule spans 300 billion)
SmolLM2-135M	135M	2 trillion
PicoLM-V3 (Instruct)	82.2M	about 757 million
PicoLM-V3 (base)	82.2M	about 757 million (unpublished pre-fine-tuning checkpoint)
PicoLM-V4-85M	85.0M	6.0 billion processed

The comparison models with a published token count saw at least 30 times as many tokens as PicoLM-V4-85M processed (OPT-125M about 30 times by corpus size, Pythia-160M 50 times, SmolLM2-135M about 330 times). The table does not control for this, and the differences in score should be read with it in mind. The released PicoLM-V3 model is the instruction-tuned one. Instruction tuning changes the output distribution and affects its scores and its loss on raw text, so the table also includes PicoLM-V3 (base): the V3 checkpoint from before fine-tuning, which was never published and was converted to the transformers format only for this comparison. Its weights differ from those of the released Instruct model by 3.2% in relative norm.

Loss on the held-out mix

The table gives mean cross-entropy in nats per token over 256 evenly spaced windows of 1024 tokens from each held-out file, computed with the same tokenizer for both models. Lower is better. See the note above on why this validation set favors PicoLM-V4-85M.

Held-out file	PicoLM-V4-85M	V3 (Instruct)	V3 (base)
Cosmopedia	1.9800	3.5770	2.4292
DCLM	3.1321	4.4505	3.7909
FineMath-3+	1.4004	2.3979	1.8398
FineWeb-Edu	2.9342	4.1570	3.3506
Python-Edu	1.4522	3.0639	2.4065
Wikipedia	2.7516	4.1749	3.3287
mean	2.2751	3.6369	2.8576

Limitations

The model is small and was trained on a small budget. It makes factual errors, contradicts itself, repeats phrases and loops in longer outputs, and writes Python that often does not run. It was trained on English text and some Python; other languages are not supported, and non-English comments in code were only partly filtered. It has no safety training, and web text can contain biased, offensive or inaccurate content that the model can reproduce. Code generation was not benchmarked. The held-out WikiText test set is drawn from Wikipedia articles, and Wikipedia is part of the training data (a 2023 dump), so WikiText perplexity may be optimistic; FineWeb-Edu, DCLM and Cosmopedia may likewise contain text that resembles benchmark items, and no decontamination against the evaluation sets was performed.

License and citation

The weights and this card are released under the Apache License 2.0. The training data come from the datasets listed above, and each of them has its own license and terms; see the individual dataset cards. Wikipedia text is licensed under CC BY-SA, and Python-Edu consists of files that keep their original open-source licenses. Anyone redistributing the model or using it commercially should check that these terms suit their use.

@misc{picolm_v4,
  title  = {PicoLM-V4-85M},
  author = {aethertp},
  year   = {2026},
  url    = {https://huggingface.co/aethertp/PicoLM-V4-85M}
}