Instructions to use Meridian21Lab/concept-memory-140m-11.5bt with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Meridian21Lab/concept-memory-140m-11.5bt with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Meridian21Lab/concept-memory-140m-11.5bt", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("Meridian21Lab/concept-memory-140m-11.5bt", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Meridian21Lab/concept-memory-140m-11.5bt with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Meridian21Lab/concept-memory-140m-11.5bt" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Meridian21Lab/concept-memory-140m-11.5bt", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Meridian21Lab/concept-memory-140m-11.5bt
- SGLang
How to use Meridian21Lab/concept-memory-140m-11.5bt with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Meridian21Lab/concept-memory-140m-11.5bt" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Meridian21Lab/concept-memory-140m-11.5bt", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Meridian21Lab/concept-memory-140m-11.5bt" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Meridian21Lab/concept-memory-140m-11.5bt", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use Meridian21Lab/concept-memory-140m-11.5bt with Docker Model Runner:
docker model run hf.co/Meridian21Lab/concept-memory-140m-11.5bt
concept-memory-140m-11.5bt: a small base model that reads 64k tokens with a concept memory
Meridian21Lab/concept-memory-140m-11.5bt ·
Project page ·
Code · Evaluation details ·
Training on W&B ·
Same data, other read: topic-index-140m-11.5bt
A 141.5M-parameter English base language model, trained from scratch on 11.5B tokens of public data for about $320. Its new part is a concept memory: while it reads, it groups the text it has already seen into small clusters of related tokens ("concepts", about 8 tokens each) and keeps a summary of each; for every next word it rereads the few clusters that matter most word for word and glances at the summaries of all the others. Each of its local layers sees only the last 128 tokens, yet the model uses up to 64k tokens of a document (about 50,000 words, a short novel).
It is the first public model of MrCogito, a project building models that compress long text into concepts and reason over them. A research release: base model, one training seed, not instruction-tuned.
Meridian21 trains several models on the same data, in the same order, for the same budget, with the same layers, and changes only the reading layer, so each card compares them on the same tests. Released so far: this model and topic-index-140m-11.5bt (Latent Sparse Attention, Kubicek et al.). The two tie on long documents (0.795 vs 0.797 bits per byte) and the seven standard tasks (mean 46.2 vs 46.5 %); topic-index-140m-11.5bt finds more facts far back (RULER mean 0.54 vs 0.44 at 8k).
Highlights
- The best long-document prediction among models of its size. On 55 held-out 64k-token documents (books, papers, government reports) it predicts the last pages at 0.795 bits per byte (lower is better; scale below). Same-size models: SmolLM2-135M 0.832 (it reads 8k tokens), Pythia-160M 0.942 (2k). Larger ones: level with Gemma 3 270M (0.805; twice our size, 520× our training text), ahead of LFM2.5-350M-Base (0.879 at its best context), behind Granite 4.0 350M (0.776).
- It uses the whole 61k-token past, and it reads it. Its predictions keep improving as the context grows to 61k tokens (0.051 bits per byte from 1k to 61k): as much as Gemma 3 270M gains up to its 32k limit, and then further. Replace the far past with another document and the gain disappears, on 55 of 55 documents.
- Strong for its training budget. Grammar (BLiMP) 80.56 %: higher than SmolLM2-135M after 252B tokens (79.72 %) and than LFM2.5-350M-Base (80.17 %; 2.5× our size, 28T tokens), level with SmolLM2's final ≈2T-token model (80.15 %). Against Pythia-160M at our budget (10.5–12.6B tokens) it is ahead on every test; Pythia does not reach our grammar or science-quiz score at any checkpoint up to 300B tokens.
- Finds a planted fact far back. RULER's needle in real essays: 1.00 up to 16k tokens and 0.98 at 32k (after training on similar lookups). At 8k it is first of six small models on needles hidden among similar keys (0.62; the others 0.20–0.59, topic-index-140m-11.5bt included).
- Cheap, open, and one line to load. ≈40 hours on one 4-GPU node (≈ $320). It loads with
AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True)and scores a whole 64k document in one pass on a single 24 GB RTX 3090; every training source is a public Hugging Face dataset, every training stage is on public W&B, and every number on this card has its raw result file in this repository.

Six standard benchmarks, same harness for all three, each score as a share of the best of the three (rim = best). This model and topic-index-140m-11.5bt trained on 11.5B tokens each, SmolLM2-135M on 252B at this checkpoint. Every number, Pythia-160M and LFM2.5-350M-Base included, in the table below.
Results at a glance
All numbers in this table come from one harness, run by us on RTX 3090 cards on 2026-11-17/18: EleutherAI's lm-evaluation-harness 0.4.11, 0-shot, all test items, max length 2,048; each model in its training precision (our two models, SmolLM2 and LFM2.5 in bf16, Pythia in fp32). Long documents: our cross-model test (below). Accuracy in %, higher is better; best in each row in bold.
| Measure | Chance | concept-memory-140m-11.5bt · 11.5B tokens | topic-index-140m-11.5bt ¹ · 11.5B | SmolLM2-135M · 252B ² | Pythia-160M · 300B ³ | LFM2.5-350M-Base · 28T ⁴ |
|---|---|---|---|---|---|---|
| Grammar (BLiMP, 67 sets) | 50 | 80.56 ± 0.14 | 80.11 | 79.72 | 77.48 | 80.17 |
| Science quiz (ARC-Easy) | 25 | 48.95 ± 1.03 | 48.40 | 61.62 | 43.64 (43.52) | 70.03 |
| Hard science quiz (ARC-Challenge) | 25 | 21.2 | 21.8 | 27.1 | 19.5 (18.8) | 38.2 |
| Sensible continuation (HellaSwag) | 25 | 29.9 | 30.1 | 32.6 | 28.4 | 37.1 |
| Science with a passage (SciQ) | 25 | 77.0 | 77.6 | 80.4 | 75.4 (74.1) | 90.1 |
| Last word of a passage (LAMBADA) | ≈0 | 33.2 | 33.3 | 35.0 | 35.4 (32.8) | 35.8 |
| Physical common sense (PIQA) | 50 | 62.8 | 63.4 | 65.9 | 62.3 (62.7) | 66.6 |
| Pronoun resolution (WinoGrande) | 50 | 50.4 | 50.7 | 50.9 | 51.2 (53.1) | 55.6 |
| Mean of the 7 tasks above | 28.6 | 46.2 | 46.5 | 50.5 | 45.1 | 56.2 |
| Wikipedia text (WikiText-2), bits per byte ↓ | — | 0.964 | 0.964 | 0.905 | 0.944 | 1.078 |
| Long documents, bits per byte ↓ (most context the model reads) | — | 0.795 (61k) | 0.797 (61k) | 0.933 (2k) | 0.942 (2k) | 0.879 (4k, its best) |
| Gain from far context, 1k → most, bits per byte ↑ | 0 | 0.051 | 0.049 | 0 (reads 2k) | 0 (reads 2k) | −0.035 (worse with more) |
| Finding facts far back (RULER at 8k, mean of 13) ↑ | ≈0 | 0.44 | 0.54 | — (reads 2k) | — (reads 2k) | 0.42 |
| Tiny-ML Leaderboard place (of 101) ⁵ | 24th | 26th | 9th | 44th (over the 150M cap) | 12th (over the 150M cap) |
¹ topic-index-140m-11.5bt, Meridian21's model on the same data: same data, order, budget and layers; its
reading layer is Latent Sparse Attention (Kubicek et al.) instead of a concept memory.
² SmolLM2-135M's earliest public training checkpoint (step 240,000, ≈252B tokens, 22× ours), the closest to our
budget a reader can check; its final ≈2T-token model is in the leaderboard table below. These early checkpoints read
2,048 tokens; SmolLM2's 8k context came in its last training phase.
³ Pythia-160M's final checkpoint, scored by us in fp32 (it loses ≈6 ARC-Easy points in bf16). In brackets:
EleutherAI's own published zero-shot results (evals/pythia-v1/pythia-160m in their GitHub repository, quoted).
⁴ LFM2.5-350M-Base (Liquid AI, March 2026), 2.5× our size; training tokens as its card reports (quoted). Its card
states a 32k context; on these documents it predicts best with 4k and worse with more (checked in fp32 and bf16, two
attention kernels, Transformers 4.57 and 5.23).
⁵ The Tiny-ML Leaderboard formula reproduced from
its page (snapshot of 2026-11-14), each of our runs placed on its own. WikiText-2 is reported there as byte perplexity:
1.950 for both of our models (bits per byte = log₂ of it).
What "bits per byte" means. The number of bits a model needs, on average, to store each byte (about one character) of text it has not seen, using its own predictions: 0 would be perfect prediction, 8 is plain text with no prediction at all, and lower is better. There is no 0-to-1 scale. On the same pages, gzip needs 2.49 bits per byte, xz 1.97 and bzip2 1.77; this model needs 0.795, so it could store those pages in about a tenth of their size. It counts bytes rather than tokens, so models with different tokenizers compare fairly. At these values a difference of 0.01 is about 1.2 %.
Against topic-index-140m-11.5bt (same tests, documents and settings): RULER 0.05–0.10 lower at every length, from tasks that ask for several facts at once and from counting words; level on the seven standard tasks (46.2 % against 46.5 %), WikiText (1.950 both) and long documents (0.795 against 0.797 bits per byte, this model better on 62 % of documents); grammar 0.45 points higher (80.56 % against 80.11 %). Paired differences with intervals: EVALUATION.md.
Long documents

Every model reads the same characters: the last ≈16 KB of each 64k-token span is the target, and the context before it is cut to the budget on the x-axis (sliding windows past a model's maximum length). Held-out documents from five public sources (Gutenberg books, arXiv papers, open-access academic books, US government publications, UK parliamentary debates); our two models trained on other documents of these sources, the others may have seen these very documents. One RTX 3090, 2026-11-18.
| Model | Size | Training tokens | Reads at most | Bits per byte ↓ | Gain, 1k → most ↑ | Another document's far past ↓ |
|---|---|---|---|---|---|---|
| concept-memory-140m-11.5bt | 141.5M | 11.5B | 64k | 0.795 | 0.051 [0.037, 0.067] | 0.832 |
| topic-index-140m-11.5bt | 141.5M | 11.5B | 64k | 0.797 | 0.049 [0.035, 0.065] | 0.834 |
| SmolLM2-135M, 252B checkpoint | 134.5M | 252B | 2k | 0.933 | — | 0.932 |
| SmolLM2-135M, final | 134.5M | ≈2T | 8k | 0.832 | 0.018 [0.014, 0.024] | 0.836 |
| Pythia-160M | 162.3M | 300B | 2k | 0.942 | — | 0.941 |
| Gemma 3 270M | 268.1M | 6T | 32k | 0.805 | 0.038 [0.029, 0.049] | 0.827 |
| Granite 4.0 350M base | 352.4M | ≈15T | 32k | 0.776 | 0.041 [0.033, 0.051] | 0.798 |
| LFM2.5-350M-Base | 354.5M | 28T | 32k | 0.879 at 4k; 0.928 at 28k | −0.035 [−0.051, −0.020] | 0.933 |
Training tokens as each card reports them (quoted); 95 % bootstrap intervals over documents. "Another document's far past": the own last 4k tokens kept, everything older taken from a different document of the same source.
- Ahead of every same-size model, on 75 % (SmolLM2-135M final, +0.038) to 98 % (SmolLM2-135M at 252B, +0.138) of documents, because it reads 8–30 times further back; level with Gemma 3 270M (+0.010 [−0.006, +0.025]) and ahead of LFM2.5-350M-Base at its best (+0.084 [0.060, 0.108]).
- It gains from far context like the 32k models trained on 500–1,300 times more text, and keeps going: 0.038 bits per byte from 1k to 28k (Gemma 3 270M 0.038, Granite 4.0 350M 0.041), then 0.013 more from 28k to 61k.
- It reads the content. With another document's far past it scores like a plain 4k cut, worse than the real past on 55 of 55 documents.
- Level with topic-index-140m-11.5bt: 0.002 bits per byte apart (this model better on 62 % of documents) and the same gain from far context; the two reads differ in how they find facts (RULER below), not in how well they predict text.
Our own tests of the read (context cut in our tokens, the far read or only its summaries switched off, 16k and 128k documents) are in EVALUATION.md. In short: switching the far read off costs 0.29 nats per token at 64k; past 64k, the training length, extra context no longer helps, so feed the last ≈64k tokens; past 64k, topic-index-140m-11.5bt holds up better (at 128k, 0.025 nats per token better with the whole document, on 13 of 15 documents).
Finding facts far back (RULER)
The standard long-context suite: 13 tasks, 100 prompts per task and length, greedy decoding, each task's own answer length; score = share of the expected answers found in the reply (0 to 1, higher is better). Every model runs through the same harness, its prompts sized in its own tokens; RTX 3090 cards, 2026-11-18. Not zero-shot for our two models: two training stages included planted-fact lookups of a similar shape.

| RULER at 8k, by task family | concept-memory-140m-11.5bt | topic-index-140m-11.5bt | SmolLM2-135M | Gemma 3 270M | Granite 4.0 350M | LFM2.5-350M-Base |
|---|---|---|---|---|---|---|
| One needle | 0.76 | 0.85 | 0.72 | 0.55 | 1.00 | 0.83 |
| Needle among distractor keys | 0.62 | 0.59 | 0.20 | 0.34 | 0.42 | 0.36 |
| Several values or keys at once | 0.33 | 0.67 | 0.28 | 0.29 | 0.70 | 0.51 |
| Count words | 0.29 | 0.45 | 0.11 | 0.34 | 0.31 | 0.05 |
| Follow variables | 0.01 | 0.00 | 0.06 | 0.00 | 0.17 | 0.13 |
| Answer questions | 0.17 | 0.23 | 0.15 | 0.15 | 0.27 | 0.31 |
| Mean of the 13 tasks, 8k | 0.44 | 0.54 | 0.30 | 0.33 | 0.54 | 0.42 |
| Mean of the 13 tasks, 4k | 0.49 | 0.54 | 0.40 | 0.48 | 0.62 | 0.47 |
| Mean of the 13 tasks, 16k | 0.37 | 0.44 | — | — | — | — |
| Mean of the 13 tasks, 32k | 0.32 | 0.39 | — | — | — | — |
- First on needles hidden among similar keys (0.62; topic-index-140m-11.5bt 0.59, the references 0.20–0.42), the task where the other models pick up a neighbouring key's value.
- Against topic-index-140m-11.5bt: −0.05 [−0.07, −0.03], −0.10 [−0.12, −0.08], −0.07 [−0.08, −0.05] and −0.07 [−0.09, −0.05] at 4k, 8k, 16k and 32k (95 % bootstrap over prompts; paired from 4k to 16k; at 32k eight of the 13 tasks are two separate 100-prompt draws, compared unpaired). topic-index-140m-11.5bt finds more where several facts are asked at once and when counting words, and is level with Granite 4.0 350M at 8k (0.54); on single needles the two tie.
- It keeps going past 8k: mean 0.37 at 16k and 0.32 at 32k, with the essay needle at 1.00 and 0.98 (the reference models were run at 4k and 8k only). Every task at every length: EVALUATION.md.

RULER by task family, 4k to 32k tokens: this model (blue) and topic-index-140m-11.5bt (orange, dashed), same harness and settings, 100 prompts per task and length (the same prompts for both from 4k to 16k).
Tiny-ML Leaderboard
The board ranks models up to 150M parameters on grammar (BLiMP), the easy science quiz (ARC-Easy) and Wikipedia text (WikiText-2 byte perplexity, lower is better), with a bonus of up to 50 % for small size. Board entries are the board's own runs (quoted, snapshot 2026-11-14); rows in italics are our runs of other models, each placed on its own.
| Place | Model | Size | Training tokens | BLiMP % ↑ | ARC-Easy % ↑ | WikiText-2 byte ppl ↓ | Quality ⁶ | Board score |
|---|---|---|---|---|---|---|---|---|
| 1 | Tiny Chick 10M V3 | 9.6M | 24.6B | 77.76 | 59.60 | 2.335 | 77.8 | 87.7 |
| 2 | Tiny Chick 10M V2.1 | 9.6M | 16.3B | 76.79 | 53.79 | 2.342 | 75.5 | 85.2 |
| 3 | Tiny Chick 10M V2 | 9.6M | 16.3B | 76.99 | 50.21 | 2.364 | 74.3 | 83.8 |
| 4 | Slayer139 1.02 | 139.3M | 24.0B | 79.22 | 65.91 | 1.862 | 81.7 | 83.4 |
| (4) | SmolLM2-135M, final | 134.5M | ≈2T | 80.15 | 64.27 | 1.799 | 81.5 | 83.2 |
| 5 | Boris-2-Think | 143.7M | 200B | 82.09 | 63.93 | 1.967 | 81.7 | 83.2 |
| 6 | Slayer149-ARC | 149.3M | 31B | 78.38 | 66.58 | 1.907 | 81.5 | 82.9 |
| 7 | Tiny Chick 10M | 9.6M | 8.3B | 75.12 | 48.53 | 2.287 | 73.3 | 82.7 |
| 8 | Slayer139 1.01 | 139.3M | 23.7B | 79.13 | 63.09 | 1.878 | 80.7 | 82.3 |
| (9) | SmolLM2-135M, 252B checkpoint | 134.5M | 252B | 79.72 | 61.62 | 1.873 | 80.4 | 82.1 |
| 9 | Glint-2 | 1.1M | ≈300B | 74.54 | 36.20 | 3.127 | 67.2 | 81.7 |
| 10 | Boris-2 | 143.7M | 200B | 81.13 | 59.76 | 1.895 | 80.2 | 81.7 |
| … | ||||||||
| 23 | GPT-S-5M | 5.2M | 25B | 72.27 | 35.69 | 2.570 | 67.4 | 77.7 |
| 24 | concept-memory-140m-11.5bt | 141.5M | 11.5B | 80.56 | 48.95 | 1.950 | 76.2 | 77.7 |
| 25 | Glint-2.3 | 1.7M | 10B | 67.30 | 36.00 | 3.051 | 64.8 | 77.6 |
| 26 | Zedev-134M-Fudas | 134M | 20.7B | 81.69 | 46.38 | 1.880 | 76.0 | 77.6 |
| (26) | topic-index-140m-11.5bt | 141.5M | 11.5B | 80.11 | 48.40 | 1.950 | 75.9 | 77.4 |
| 27 | Hydrion-v1-Base | 114.1M | ≈2.5B | 80.08 | 47.26 | 2.040 | 75.2 | 77.4 |
| … | ||||||||
| (44) | Pythia-160M (over the 150M cap) | 162.3M | 300B | 77.48 | 43.64 | 1.924 | 73.5 | 74.5 |
⁶ Quality: the mean of BLiMP, ARC-Easy and the board's WikiText score, before the size bonus. By quality alone the model is 14th of 101; on grammar alone, 8th. Of the 55 board entries that report 15B training tokens or fewer, three place above it. topic-index-140m-11.5bt places 26th (16th by quality). Full board with our rows: EVALUATION.md.
Data efficiency
"Data-efficient" here means a better score from the same number of training tokens, measured against the published training checkpoints of Pythia-160M (one every 2.1B tokens) and SmolLM2-135M (from 252B tokens), all scored by us with the same harness.

- Same budget: against Pythia-160M at 10.5B and 12.6B tokens, this model is ahead on every test: +2.3 to +2.6 points of grammar, +6.2 to +6.4 on the science quiz, +3.0 to +4.1 on the 7-task mean, and 0.07–0.09 lower WikiText byte perplexity.
- Tokens to match: Pythia-160M does not reach this model's grammar or science-quiz score at any checkpoint up to 300B tokens (26× ours); it reaches its WikiText score after ≈36B tokens (3.1×). SmolLM2-135M stays below its grammar score at 252B and 503B tokens and is level at ≈2T (80.15 %); it is ahead on the knowledge tests from its first checkpoint.
- What it does not separate: architecture from data. Part of the gap to Pythia is newer, cleaner data; topic-index-140m-11.5bt, on the same data and order, lands within half a point on every test; a standard transformer of the same size on our data and budget (training) will measure the architecture's share.
What its text looks like
Nucleus sampling (temperature 0.7, top-p 0.9, repetition penalty 1.1), 60 new tokens, seed 0, not picked; prompt
in bold, unedited, cut at a sentence end (…). More seeds and beam-search outputs: samples/samples_ours.json.
The capital of France is Paris, the second largest city in the world, with a population of 7.5 million people. It is known for its rich history and culture, as well as its iconic landmarks such as the Eiffel Tower, Louvre Museum, and Notre-Dame Cathedral. …
In 1905, Albert Einstein published his famous theory of general relativity, which was later later developed by Wolfgang Pauli. …
def fibonacci(n):
"""Returns the nth Fibonacci number""" if len(fibonacci) == 0: return 0
Fluent, on-topic English and code-shaped output; the facts are often wrong (Paris has ≈2.1 million people; the 1905 paper was special relativity), as expected from a 140M base model at 11.5B tokens.
How the concept memory works
Think of a student reading a long book for an exam. Rereading every page for every answer is what full attention does: exact, but the cost grows with the square of the book's length. Our student keeps a notebook instead.
tokens ─► 3 local layers ─► ┌──── looped 4 times, shared weights ────┐ ─► 8 local layers ─► next token
(last 128 tokens) │ memory-reading layer ─► 1 local layer │ (last 128 tokens)
└────────────────────────────────────────┘
memory-reading layer, for each new token, one softmax over:
the document's first 4 tokens + the current and previous 512-token block, word for word
+ up to 512 tokens from the concepts that matter most, word for word ("opened")
+ one summary entry for every other concept (64 concepts per closed 512-token block)
Concepts are clusters of similar keys in each closed 512-token block (a balanced k-means, no learned parameters), kept with a count, mean key, mean value and spread. From those, each new token estimates which clusters matter, opens them and reads the rest as summaries, so every past token counts exactly once. The design's small "reasoner" over the concepts is switched off in this release, which tests the memory only.
Training
| Parameters | 141,506,208 (input and output share one token table, counted once); 122.6M without the 49,152 × 384 table |
| Shape | width 896; 13 distinct layers; one memory-reading layer; local layers see the last 128 tokens |
| Tokenizer | SmolLM2's (HuggingFaceTB/SmolLM2-135M), 49,152 tokens; every document starts with `< |
| Budget | 11.5B tokens in three length stages: 8.0B at 4k-token rows → 3.0B at 16k → 0.5B at 64k |
| Optimizer | Muon (peak 0.01) for the weight matrices, AdamW for the rest; bf16; seed 42 |
| Machine | one AWS spot node with 4 × NVIDIA L40S (48 GB) |
| Time and cost | ≈40 hours (≈160 GPU-hours) including ≈1 h of tests; ≈ $320 at ≈ $8/h spot, plus $15 for an optimizer test; energy not measured |
| Finished | 2026-11-16 23:16 UTC |
Data (m21mix_v2), all public Hugging Face datasets. Shares are of each stage's tokens.
| Stage | Tokens | What it reads |
|---|---|---|
| 1. Main, 4k rows, documents packed | 8.0B | Nemotron-CC v2 high-quality web 30 % and its Diverse-QA rewrites 22 %, DCLM-Edu (score ≥ 3) 12 %, educational code (Stack v2 Edu) 10 %, Nemotron Wikipedia rewrites 8 %, Nemotron-CC-Math (4+) 8 %, Nemotron reading-comprehension QA 6 %, Nemotron math textbooks 4 % |
| 2. Decay, 16k rows, one document each | 3.0B | long documents 60 %: science PDFs (Dolma 3 Longmino) 22 %, Gutenberg books 12 %, code packed per repository 12 %, arXiv 5 %, open-access books (DOAB) 3 %, US government documents 3 %, UK Hansard 3 %; web replay 20 %; Nemotron InfiniByte reasoning 7 %; multi-hop "libraries" 6 %; planted-fact lookups 3 %; chat (smol-smoltalk) 4 % |
| 3. Long, 64k rows, one document each | 0.5B | science PDFs 30 %, books 25 %, code repositories 15 %, multi-hop libraries 10 %, lookups 5 %, web replay 5 %, arXiv 4 %, DOAB 3 %, US GPO 3 % |
The two constructed row types (about 3 % of all tokens): multi-hop libraries, the paragraphs of 20–160 2WikiMultihopQA training questions shuffled, with the questions and answers at the end; and lookups, 4–24 invented facts planted in a real book passage and asked at the end. The test sets on this card (BLiMP, ARC, WikiText, HotpotQA, SQuAD, the held-out long documents) are not in the training data. Stage-by-stage settings and losses: EVALUATION.md.
Scope and trade-offs
- Base model. No instruction or preference tuning; about 1 % of training text was chat transcripts rendered as plain text (the tokenizer folder carries a ChatML template for that reason). English, plus code.
- Knowledge tests (ARC, SciQ, HellaSwag, LAMBADA) are below SmolLM2-135M, which trained on 22× (at 252B) to ≈170× (final) more text, and LFM2.5-350M-Base (2.5× our size, ≈2,400× our text); at this size these tests track training tokens closely.
- Next to topic-index-140m-11.5bt: grammar 0.45 points higher (80.56 % against 80.11 %, about two standard errors); level on the other standard tests and on long documents (0.002 bits per byte ahead); RULER 0.05–0.10 lower at every length; past 64k topic-index-140m-11.5bt holds up better (0.025 nats per token at 128k).
- Context: trained and tested to 64k tokens. Past 64k, extra context no longer improves predictions (at 128k, the whole document is 0.06 nats per token worse than its last 61k, on 15 of 15 documents); feed the last ≈64k.
- Hard RULER tasks: variable tracking stays at 0.00–0.01 at every length, the UUID tasks at 0.04–0.29, and most tasks fall with length; tasks that need several values or keys at once score lower than single lookups.
- Controls not run yet: one seed; no full-attention model of the same size on the same data, so the share of the architecture in any result is not separated from the data.
- Facts and safety: it makes things up, and has no safety tuning or output filtering; not for high-stakes or user-facing use.
What's next
- A standard transformer of the same size on the same data and budget (training since 2026-11-18), to measure how much of the long-context gain belongs to the read; and a third reading layer, the catalogue, on the same data (training).
- A second seed for both 150M models, so that differences of a point or two can be called.
- Longer budgets and larger models with the better read, and reasoning over concepts (built, switched off in both releases).
How to use
Needs torch and transformers (tested with 4.57.6 and 5.23.0). The modeling code ships in this repository, so
loading takes trust_remote_code=True. Runs on one CUDA GPU in bf16.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "Meridian21Lab/concept-memory-140m-11.5bt"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to("cuda")
inputs = tok("The capital of France is", return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=40, do_sample=True, temperature=0.7, top_p=0.9,
repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))
# scoring: mean next-token loss in nats per token
loss = model(**inputs, labels=inputs["input_ids"]).loss
generateis 🤗 Transformers' own (sampling, greedy, beam search), running over the model's cached decoder. This is a base model, not a chat model.- Long documents (up to the 64k training length): add
attn_backend="flex"tofrom_pretrained(PyTorch flex attention on CUDA). The default,sdpa, builds a dense attention mask and suits short inputs. - For exact scores, batch prompts of one length or score them one at a time: left padding moves the reading layer's block edges and changes scores slightly.
modeling_perceiver_ar.pyis generated from the MrCogito source byscripts/export_hub_model.py;export_manifest.jsonnames the source commit. Our lm-eval adapter, the long-document scripts inlong_context/and the sampling script that wrotesamples/run from the MrCogito code at1fddc505.
Citation
@misc{sopyla2026conceptmemory150m,
author = {Sopyła, Krzysztof},
title = {concept-memory-140m-11.5bt: a small base language model with a long-context concept memory},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/Meridian21Lab/concept-memory-140m-11.5bt}},
note = {Meridian21 Lab, MrCogito project. Code: \url{https://github.com/ksopyla/MrCogito}}
}
@online{sopyla2026mrcogito,
author = {Sopyła, Krzysztof},
title = {MrCogito: Concept Reasoning Model},
year = {2026},
url = {https://ai.ksopyla.com/projects/concept-reasoning/},
urldate = {2026-11-18}
}
Licence
Weights and card: Apache License 2.0. The training data come from public Hub datasets under their own licences: DCLM-Edu (CC BY 4.0), Dolma 3 Longmino (ODC-By), smol-smoltalk (Apache-2.0), the openly licensed Common Pile sources, Nemotron-Pretraining-Specialized (CC BY 4.0, CC BY-SA 4.0, GFDL and NVIDIA terms per subset), and NVIDIA's Nemotron-CC-v2 and Nemotron-CC-Math-v1 (gated on the Hub under NVIDIA's terms; about 45 % of the training tokens). See each dataset's card for its terms.
Changelog and contact
- 2026-11-18: first public release (weights, card, evaluation page, raw results, figures).
- 2026-11-18: loads with 🤗 Transformers
from_pretrained(modeling code,generation_config.jsonandexport_manifest.jsonadded;config.jsongainsauto_map). - 2026-11-18: RULER against four reference models at 4k and 8k; link to topic-index-140m-11.5bt; note that RULER's 32k prompts differ between runs for 8 of 13 tasks.
- 2026-11-18: aligned with the topic-index-140m-11.5bt card: topic-index-140m-11.5bt in every results table and chart, one colour per model across both cards, a head-to-head section in EVALUATION.md.
Krzysztof Sopyła · ai.ksopyla.com · Meridian21 Lab · questions and errors: open a discussion on this page.
- Downloads last month
- 681
Datasets used to train Meridian21Lab/concept-memory-140m-11.5bt
HuggingFaceTB/dclm-edu
common-pile/stackv2_edu_filtered
Evaluation results
- accuracy (%) on BLiMPself-reported80.560
- accuracy (%) on ARC-Easytest set self-reported48.950
- byte perplexity (lower is better) on WikiText-2self-reported1.950