nazdef/1gpu-llm-small-en-it-base-v2

1gpu-llm Small EN/IT Base v2

This repository is the promoted v2 base release for the 1gpu-llm small EN/IT family.

1gpu-llm is a family of language models trained on a single consumer GPU. For this small line, the reference hardware is:

• GPU: NVIDIA GeForce RTX 4060 Ti 16GB
• training setup: single GPU
• model family: GPT-2-style decoder with pre-layernorm blocks

Concretely, this release promotes the post-decay winner at step_18800:

• repo id: nazdef/1gpu-llm-small-en-it-base-v2
• released checkpoint: step_18800.pt
• role: best promoted small-family base checkpoint so far
• languages: English + Italian
• context window: 2500 tokens
• architecture: architecture: gpt2, block_type: gpt2_prelayernorm
• parameter count: 136,128,000 parameters (~136.128M)

This is a base model, not an instruction-tuned chat model.

What Changed vs v1

The previous public base release was:

• nazdef/1gpu-llm-small-en-it-base
• released checkpoint: step_8600.pt

This v2 release does not come from a fresh scratch rerun. It comes from a two-stage continuation of that base:

1. continual pretraining from step_8600 at a lower LR
2. a short decay-only tail starting from the best pre-decay checkpoint

The promoted path is:

• v1 base source: step_8600
• continual-pretraining winner: step_18000
• decay-only promoted winner: step_18800

So this release should be read as:

the original 1gpu-llm-small-en-it-base plus a successful lower-LR continual-pretraining branch, followed by a useful short decay-only cleanup.

Provenance

• source public base repo:
  • nazdef/1gpu-llm-small-en-it-base
• source public base checkpoint:
  • step_8600.pt
• source parent run:
  • 20260622_resume-gpt2small-gpt2preln-k20-wsds800-final2e5-webwiki-step8000-dense50
• continual-pretraining run:
  • 20260701_continual-pretraining-gpt2small-step8600-lr5e5-w500-s10000-d1000-final1e5-webwiki
• decay-only tail run:
  • 20260703_resume-gpt2small-gpt2preln-k20-wsddecayonly-cpt8600-lr5e5-step18000-final5e6-webwiki
• promoted release checkpoint:
  • step_18800.pt

Training Data

This release stays on the same bilingual EN/IT web + wiki data family used by the first public small base:

• dataset id on disk:
  • 202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M
• context window during training: 2500
• packing length: 2500
• mixing strategy: source_balanced
• validation ratio: 0.05

Train-split token inventory from the dataset summary:

• train tokens: 6,899,597,399
• English train tokens: 3,593,711,492
• Italian train tokens: 3,305,883,508

Main source groups:

• English FineWeb-HQ (epfml/FineWeb-HQ)
• Italian FineWeb2-HQ (epfml/FineWeb2-HQ)
• English Wiki40B (google/wiki40b)
• Italian Wiki40B (google/wiki40b)

Continual-Pretraining Parameters

The continual-pretraining branch that starts from v1 used:

• resume source:
  • step_8600.pt
• resume mode:
  • continual_pretraining
• learning rate:
  • 5e-5
• schedule:
  • warmup: 500
  • stable: 10000
  • decay: 1000
• branch peak:
  • step_18000

Then the final cleanup used a decay-only tail:

• resume source:
  • step_18000.pt
• resume mode:
  • model_only
• schedule:
  • wsd-decay-only
• LR path:
  • 5e-5 -> 5e-6
• planned decay span:
  • 1800 steps
• promoted winner inside that tail:
  • step_18800

Important detail:

• the final release winner happened early inside the decay tail
• step_18800 beat later tail checkpoints, including step_19800

Token Accounting

Training math:

• sequence length: 2500
• batch size: 6
• grad accumulation: 16
• tokens per optimizer step: 240,000

Approximate token exposure:

• v1 base step_8600:
  • 2.064B tokens total
• extra continual-pretraining tokens from 8600 -> 18000:
  • 2.256B tokens
• extra decay-tail tokens from 18000 -> 18800:
  • 192M tokens
• total extra tokens vs v1:
  • 2.448B tokens
• total tokens seen by step_18800:
  • 4.512B tokens

So the promoted v2 base saw about 2.19x the total token exposure of the first public small base.

Why step_18800 Was Chosen

Selection happened in three layers:

1. step_8600 was the original public base
2. the lower-LR continual-pretraining branch produced a new winner at step_18000
3. the decay-only tail improved that winner again, with the best checkpoint at step_18800

Main benchmark metric:

• val_loss_mixed (lower is better)

Key checkpoints:

checkpoint	role	val_loss_mixed
step_8600	public base v1	4.7964
step_18000	continual-pretraining winner	4.7358
step_18800	promoted v2 base	4.7248

Practical deltas:

• 18800 vs 18000:
  • about -0.0111
• 18800 vs 8600:
  • about -0.0716

Direct Comparison vs the First Public Base

Main metrics

checkpoint	val_loss_en	val_loss_it	val_loss_mixed	ppl_mixed	cloze_it_contains	loop_rate	repeated_4gram_rate
step_8600	4.8075	3.6415	4.7964	121.0694	0.20	0.525	0.925
step_18800	4.6697	3.7627	4.7248	112.7033	0.26	0.375	0.875

Where v2 is better

• val_loss_mixed: 4.7964 -> 4.7248
• val_loss_en: 4.8075 -> 4.6697
• ppl_mixed: 121.07 -> 112.70
• loop_rate: 0.525 -> 0.375
• repeated_4gram_rate: 0.925 -> 0.875
• cloze_it_contains: 0.20 -> 0.26

Short honest read:

• v2 is cleaner on the main benchmark
• v2 is less loopy
• v2 is better on the mixed objective
• but v2 is not a universal win on every Italian-side scalar metric

Main remaining tradeoff

• val_loss_it is still better on v1:
  • 3.6415 on step_8600
  • 3.7627 on step_18800

So this promotion is real, but not a “literally every metric improved” fairy tale.

Probe Snapshot and Rank/Probability Readout

These probes are included because scalar loss alone is not enough.

v1 base step_8600

• The capital of Italy is -> expected Rome
  • correct_token_rank = 7
  • correct_token_probability = 0.01446533203125
  • perplexity_on_target_sequence = 69.13080168776372
• A small language model should -> expected be
  • correct_token_rank = 1
  • correct_token_probability = 0.46484375
  • perplexity_on_target_sequence = 2.1512605042016806
• La capitale d'Italia è -> expected Roma
  • correct_token_rank = 6
  • correct_token_probability = 0.0257568359375
  • perplexity_on_target_sequence = 38.824644549763036
• Un piccolo modello linguistico dovrebbe -> expected essere
  • correct_token_rank = 1
  • correct_token_probability = 0.4453125
  • perplexity_on_target_sequence = 2.245614035087719

v2 base step_18800

• The capital of Italy is -> expected Rome
  • correct_token_rank = 8
  • correct_token_probability = 0.015869140625
  • perplexity_on_target_sequence = 63.01538461538459
• A small language model should -> expected be
  • correct_token_rank = 1
  • correct_token_probability = 0.51171875
  • perplexity_on_target_sequence = 1.9541984732824427
• La capitale d'Italia è -> expected Roma
  • correct_token_rank = 10
  • correct_token_probability = 0.01513671875
  • perplexity_on_target_sequence = 66.06451612903226
• Un piccolo modello linguistico dovrebbe -> expected essere
  • correct_token_rank = 1
  • correct_token_probability = 0.404296875
  • perplexity_on_target_sequence = 2.473429951690821

Practical read:

• procedural prompts remain healthy and top-1
• English procedural confidence improved
• Italian factual probing is not cleaner in this snapshot
• this matches the broader story:
  • v2 is better as a benchmark checkpoint
  • but factual/cloze behavior, especially in Italian, still needs adult supervision

Decoding Recommendation

A dedicated comparative GPU decoding sweep was run across:

• step_8600
• step_18000
• step_18800

The outcome split in two:

• checkpoint-level winner:
  • step_18800
• best decoding-grid pairing overall:
  • step_18000 + anti_loop

For the promoted v2 base, this repo ships a safety-first default:

• recommended preset:
  • anti_loop
• alternate preset worth trying:
  • creative

Why anti_loop as the public default:

• step_18800 is the best checkpoint overall
• its own tuning/holdout sweep is split:
  • tuning prefers anti_loop
  • holdout prefers creative
• anti_loop is the conservative choice for public default behavior

Recommended generation params:

• do_sample = true
• temperature = 0.8
• top_k = 50
• top_p = 0.9
• repetition_penalty = 1.15
• no_repeat_ngram_size = 4
• max_new_tokens = 64

Both generation_config.json and recommended_decoding_params.json are included in the repo.

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

repo_id = "nazdef/1gpu-llm-small-en-it-base-v2"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "A small language model should"
prompt_ids = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
bos = torch.tensor([[tokenizer.bos_token_id]], dtype=prompt_ids["input_ids"].dtype)
input_ids = torch.cat([bos, prompt_ids["input_ids"]], dim=1)
attention_mask = torch.ones_like(input_ids)

outputs = model.generate(
    input_ids=input_ids,
    attention_mask=attention_mask,
    do_sample=True,
    max_new_tokens=64,
    temperature=0.8,
    top_k=50,
    top_p=0.9,
    repetition_penalty=1.15,
    no_repeat_ngram_size=4,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Files Included

• original .pt checkpoint
• exported checkpoint-native .safetensors weights plus metadata sidecar
• standard Transformers model.safetensors
• Transformers config.json
• tokenizer files
• continual-pretraining config
• decay-only config
• decay-run telemetry (best_validation.json, metrics.jsonl, eval_metrics.jsonl, probe_generations.jsonl)
• continual-pretraining closeout report
• decay-tail closeout report
• cross-checkpoint decoding-grid report
• probe summary probe_step18800_summary.json
• recommended generation settings (generation_config.json, recommended_decoding_params.json)
• release note release_note.md

Intended Use

Use this model as:

• the promoted small bilingual base checkpoint v2 of the 1gpu-llm family
• a stronger loss/benchmark starting point than v1
• a base model for further evaluation or finetuning

Do not read this release as:

• an instruction-tuned assistant
• a claim that all factual Italian behavior is solved
• a reason to ignore decoding settings

License

This release is published under CC-BY-SA-4.0.

The training data mix includes:

• FineWeb-HQ / FineWeb2-HQ web data
• Wiki40B English + Italian

Downstream users are responsible for verifying that their intended use complies with the relevant upstream dataset terms and with share-alike obligations where they apply.