Qwen3-1.7B on Nemotron-CC (low-quality tier): ablation checkpoints

Four base language models with the Qwen3-1.7B architecture (1.72B parameters). Each was pretrained from scratch for one epoch on 34.4B tokens of the low-quality (LQ) tier of Nemotron-CC, Common Crawl snapshots 2013–2015.

Folder Training data Seed
v2-lq-seed1234/ LQ after V2 repair (prose-only repair + linguistic repair) 1234
raw-lq-seed1234/ LQ, unrepaired 1234
raw-lq-seed1337/ LQ, unrepaired 1337
raw-lq-seed42/ LQ, unrepaired 42
  • Seeds: the seed sets the weight initialization only. Every run used the same data-shuffling seed (1234), so the three raw runs saw identical batches in identical order. v2-lq-seed1234/ and raw-lq-seed1234/ share the same configuration and seed and differ only in training data.
  • Training: 16,406 steps × 512 sequences × 4,096 tokens, BF16, AdamW (β = 0.9/0.95, weight decay 0.1), learning rate 3e-4 → 3e-5 (warmup–stable–decay).
  • Base models only: no instruction tuning.

Format and compatibility

Each folder is a standard Hugging Face checkpoint: BF16 model.safetensors, tokenizer, and generation config. The checkpoints were exported with transformers 5.3. config.json and tokenizer_config.json also carry the older field names, so transformers 4.x (4.51 or newer) loads the same RoPE base (1,000,000) and the same tokenizer.

Usage

hf download anonymous-md/test --include "v2-lq-seed1234/*" --local-dir ./qwen3-lq
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "./qwen3-lq/v2-lq-seed1234"
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path)
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support