Sapling-505M-Base

Sapling is a 505M-parameter English base language model trained from randomly initialized weights. It uses the Qwen3 dense architecture with its own 32,768-entry tokenizer, and it was pretrained on 5.0 billion tokens: 80% FineWeb-Edu and 20% Cosmopedia v2.

This is a base text-completion model. It continues text. It does not follow instructions or chat. An instruction-tuned version may follow later.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "VinayRevanuru/Sapling-505M-Base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto")

inputs = tok("The best way to cook rice is", return_tensors="pt")
out = model.generate(
    **inputs, max_new_tokens=80,
    do_sample=True, temperature=0.8, top_p=0.9, repetition_penalty=1.1,
)
print(tok.decode(out[0], skip_special_tokens=True))

Use sampling. Greedy decoding loops quickly at this scale (see Limitations).

Model details

Architecture	Qwen3ForCausalLM (dense): GQA, QK-Norm, RMSNorm, RoPE, SwiGLU
Parameters	505,483,776 (embeddings tied)
Layers / width / FFN	30 / 1,024 / 3,072
Attention heads	16 query, 8 key/value, head dim 128
Context length	2,048 tokens
RoPE θ	1,000,000
Vocabulary	32,768 byte-level BPE, trained from scratch (`<
Weights	FP32 safetensors (≈2.0 GB)

Where the parameters go: feed-forward 283.1M (56%), attention 188.7M (37%), tied embedding 33.6M (7%).

Training data

The mix is 80/20 by consumed training tokens, counted with Sapling's own tokenizer:

• FineWeb-Edu (sample-10BT): filtered educational web text.
• Cosmopedia v2 (smollm-corpus, cosmopedia-v2): synthetic textbook-style text, text field only.

How the data was prepared:

• About 0.1% of documents per source were held out by a deterministic hash before tokenizer training and packing.
• Documents were joined with EOS separators and packed into full 2,048-token sequences, with no padding or truncation.
• Sequences were stored in 52.4M-token shards and read in a deterministic 4:1 FineWeb:Cosmopedia interleave.
• Every shard was seen at most once.

Training procedure

Tokens seen	5,000,396,800 (76,300 optimizer steps)
Global batch	65,536 tokens (32 × 2,048)
Optimizer	fused AdamW, β = (0.9, 0.95), ε = 1e-8, weight decay 0.1 (norms/biases excluded)
Learning rate	peak 3e-4, 1% linear warmup, cosine decay
Gradient clipping	1.0
Precision	BF16 autocast, FP32 master weights
Hardware	8 × NVIDIA A100 80GB (PyTorch DDP) on Modal
Throughput / wall clock	≈142k tokens/s, ≈10.1 h (≈81 GPU-hours)
Software	torch 2.6.0, transformers 4.51.3, CUDA 12.4

Note on the schedule: the cosine was configured for a 7B-token run, and training was stopped early at 5B. This checkpoint therefore has not been fully annealed: the learning rate at the final step was about 8.2e-5, against a planned floor of 3e-5. A short cooldown would likely lower its loss somewhat.

Training loss fell from 10.2 (random init; ln 32768 ≈ 10.4) to about 2.75 (100-step average). The repeating dips in the curve are Cosmopedia shards, which are easier to predict than web text. Median single-step loss in the last block of each source:

Source	Loss at ~0.3B tokens	Loss at ~5.0B tokens
FineWeb-Edu	3.45	2.83
Cosmopedia v2	2.59	1.72

These are training losses on unseen data, since every shard is read once. Held-out validation losses were not recorded for this checkpoint.

Sample generations (5B checkpoint, temperature 0.8, top-p 0.9)

The best way to cook rice is to boil it, which is a great way to prepare a whole meal that is often enjoyed as a side dish or as a main course. If you're trying to make a quick breakfast, you can make a simple rice pudding. Simply mix together rice, eggs, milk…

The capital of France is Paris, located in the North-West of France. France is a country of great culture and history, with a rich heritage of art, music,eff, and cuisine…

The largest planet in our solar system is Mercury, which is a gas giant planet, about the size of Earth…

The last example is included deliberately. Fluent, confident and wrong is typical of a 0.5B base model.

Limitations

• Not factually reliable. It produces fluent but often false statements: wrong boiling points, wrong planets, invented events.
• Repetition. Greedy decoding often falls into loops. Use sampling, and a repetition penalty helps.
• Weak at code and math. The training data contained little code, and arithmetic is unreliable.
• English only, with a 2,048-token context.
• No safety tuning. It can reproduce biases and errors present in web and synthetic text. Do not use it for decisions about people or in any high-stakes setting.

Intended use

Research and education on small-model pretraining, a starting point for instruction tuning or LoRA experiments, and a baseline for tokenizer and data-mix ablations.

training_summary.json in this repo records the full configuration, data manifest, tokenizer sha256 (f7d39923…0be0984), and dependency versions.