exnivo/tinybrain-100m-base

TinyBrain-100M Base

A 103M parameter English causal language model trained from scratch.

TinyBrain-100M Base is a small LLaMA-style causal language model trained from scratch on the exnivo/tinybrain-pretrain-corpus-2b dataset.

This is a base model, not an instruct/chat model. It is intended for language modeling experiments, continued pretraining, supervised fine-tuning, and small-model research.

For chat or instruction-following behavior, use the instruction-tuned version:

exnivo/tinybrain-100m-instruct

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "exnivo/tinybrain-100m-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "Photosynthesis is the process by which plants"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    temperature=0.8,
    top_p=0.9,
    repetition_penalty=1.08,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

At a Glance

Item	Details
Model type	Base causal language model
Parameters	103,385,856
Approx. size	103.4M
Architecture	LLaMA-style causal transformer
Language	English
Context length	2048 tokens
Vocabulary size	24,000
Tokenizer	Custom TinyBrain tokenizer
Training style	From scratch pretraining
Pretraining dataset	exnivo/tinybrain-pretrain-corpus-2b
Instruct dataset	exnivo/tinybrain-instruct-sft-200k
Instruct model	exnivo/tinybrain-100m-instruct

Model Details

Item	Value
Parameters	103.4M
Architecture	llama / LlamaForCausalLM
Vocabulary size	24,000
Context length	2048 tokens
Hidden size	768
Intermediate size	2048
Layers	12
Attention heads	12
Key/value heads	12
Activation	SiLU
RMS norm epsilon	1e-05
Tied embeddings	true
BOS token	`<
EOS token	`<
PAD token	`<
Dataset	exnivo/tinybrain-pretrain-corpus-2b

Intended Use

TinyBrain-100M Base is intended for:

• small language model research
• causal language modeling experiments
• continued pretraining
• supervised fine-tuning
• instruction tuning
• tokenizer/model experiments
• educational small-model projects
• comparing base vs instruct behavior
• lightweight local model experiments

This model is best used as a base checkpoint for further training.

Not Intended For

This model is not intended to be used directly as a finished assistant.

Do not rely on the base model for:

• polished chat behavior
• instruction following
• safety-critical answers
• factual authority
• medical, legal, or financial advice
• live/current information
• advanced reasoning
• production use without evaluation

For assistant-style behavior, use exnivo/tinybrain-100m-instruct instead.

Training Data

TinyBrain-100M Base was trained on:

exnivo/tinybrain-pretrain-corpus-2b

The pretraining corpus is a mixed-source English dataset containing factual text, educational text, math reasoning data, code data, conversation-style data, and clean web text.

The pretraining corpus scan found:

Metric	Value
Rows	3,013,308
Characters	7,767,447,861
Words	1,249,832,587
Approx. tokens	~1.81B tokenizer-independent estimate

The training run used an estimated ~2.1B training tokens. Token counts may differ depending on tokenizer, packing, filtering, and training pipeline details.

Dataset Mix

The pretraining corpus includes these broad categories:

Category	Rows	Percent
factual	773,492	25.67%
educational	752,625	24.98%
math_reasoning	633,341	21.02%
code	326,019	10.82%
conversation	296,728	9.85%
clean_web	231,103	7.67%

Relationship to TinyBrain

TinyBrain is a small LLM project focused on compact datasets, small base models, and instruction-tuned models.

Stage	Repository	Purpose
Pretraining corpus	exnivo/tinybrain-pretrain-corpus-2b	Base language model training data
Base model	exnivo/tinybrain-100m-base	Small causal LM trained from scratch
SFT dataset	exnivo/tinybrain-instruct-sft-200k	Instruction/chat fine-tuning data
Instruct model	exnivo/tinybrain-100m-instruct	Chat/instruct model fine-tuned from the base model

Pipeline:

TinyBrain Pretrain Corpus 2B
        ↓
TinyBrain-100M Base
        ↓
TinyBrain Instruct 200K
        ↓
TinyBrain-100M Instruct

Evaluation

A quick WikiText-2 evaluation was run on the base model.

Metric	Value
Eval tokens	38,138
Eval text chars	159,791
Loss	3.7440
Perplexity	42.27

This is a lightweight evaluation, not a full benchmark suite. Results may vary depending on evaluation script, tokenizer settings, context length, and dataset preprocessing.

Base Model Behavior

TinyBrain-100M Base is a raw pretrained model. It can complete text, but it is not tuned to follow instructions.

Example base prompt:

Photosynthesis is the process by which plants

The base model may continue with partially useful text, but it can also repeat, drift, hallucinate, or produce broken completions. This is expected for a small base model and is one reason instruction tuning is needed.

For better chat behavior, use:

exnivo/tinybrain-100m-instruct

Recommended Generation Settings

For raw base-model text completion:

temperature = 0.8
top_p = 0.9
max_new_tokens = 80
repetition_penalty = 1.08

For more stable completions:

temperature = 0.5
top_p = 0.85
max_new_tokens = 80
repetition_penalty = 1.1

For deterministic testing:

do_sample = False
max_new_tokens = 80

Example: Text Completion

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "exnivo/tinybrain-100m-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "Gravity is the force that"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.9,
    repetition_penalty=1.08,
    pad_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Example: Fine-Tuning Starting Point

TinyBrain-100M Base can be fine-tuned on the TinyBrain SFT dataset:

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from trl import SFTTrainer, SFTConfig

base_model = "exnivo/tinybrain-100m-base"
dataset_id = "exnivo/tinybrain-instruct-sft-200k"

tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(base_model)

ds = load_dataset(dataset_id, split="train")

def format_example(example):
    text = ""

    for message in example["messages"]:
        role = message["role"]
        content = message["content"].strip()

        if role == "user":
            text += f"User: {content}\n"
        elif role == "assistant":
            text += f"Assistant: {content}\n"

    return {"text": text.strip()}

ds = ds.map(format_example)

config = SFTConfig(
    output_dir="tinybrain-100m-instruct-sft",
    dataset_text_field="text",
    max_seq_length=512,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    num_train_epochs=1,
    logging_steps=20,
    save_steps=500,
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=ds,
    args=config,
)

trainer.train()

Training

TinyBrain-100M Base was trained from scratch on the TinyBrain pretraining corpus.

Training details:

Item	Value
Training type	From-scratch causal language modeling
Dataset	TinyBrain Pretrain Corpus 2B
Approx. training tokens	~2.1B
Reported best validation loss	2.6779
Training precision	bf16
Hardware	NVIDIA RTX PRO 6000 Blackwell Server Edition

Strengths

TinyBrain-100M Base is useful because it is:

• small and lightweight
• trained from scratch
• easy to inspect
• easy to fine-tune
• based on an open TinyBrain data pipeline
• trained on a compact mixed-source corpus
• suitable for small-model experiments
• useful as a base checkpoint for SFT

Limitations

TinyBrain-100M Base has important limitations.

The model may:

• hallucinate facts
• produce broken or repetitive text
• fail at math
• fail at instruction following
• misunderstand prompts
• generate incomplete code
• produce outdated or incorrect information
• drift off-topic
• repeat web/data artifacts

This is expected for a small base model. It has not been tuned to reliably follow user instructions.

For chat and assistant behavior, use the instruction-tuned model instead.

Suggested Evaluation

Recommended checks:

• validation loss / perplexity
• text completion quality
• repetition behavior
• short factual completions
• simple math completions
• code completion sanity checks
• hallucination checks
• before/after SFT comparison
• downstream instruction-following after fine-tuning

Example base-model prompts:

Paris is the capital city of

The Netherlands is a country in

A cat is an animal that

One plus one equals

Photosynthesis is the process by which plants

Citation

If you use this model, you can cite it as:

@misc{tinybrain_100m_base,
  title = {TinyBrain-100M Base},
  author = {exnivo},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/exnivo/tinybrain-100m-base}}
}

Related Repositories

• Pretraining corpus: exnivo/tinybrain-pretrain-corpus-2b
• Base model: exnivo/tinybrain-100m-base
• SFT dataset: exnivo/tinybrain-instruct-sft-200k
• Instruct model: exnivo/tinybrain-100m-instruct

License

This model is released under the Apache 2.0 license.

The training dataset is mixed-source and currently listed under license: other. Users should review the upstream dataset licenses and source metadata before commercial use of models trained or fine-tuned from this checkpoint.

Disclaimer

TinyBrain-100M Base is an experimental small base language model. It may produce incorrect, biased, unsafe, nonsensical, or misleading outputs.

Do not use it for high-stakes applications without additional training, filtering, evaluation, and safeguards.