pollock-mini-lm-125m / CHANGELOG.md
dawidmajewski's picture
Release Pollock 1.0 (r002)
5e571cd verified
|
Raw
History Blame Contribute Delete
1.08 kB

Changelog

Revision numbers identify published model states independently of release names.

r002 — Pollock 1.0

  • Expanded the architecture from 12/12/768 to 12/14/896 while remaining below 128M total parameters.
  • Reduced context from 2,048 to 1,024 tokens and increased the effective batch from 262,144 to 491,520 tokens.
  • Changed the learning-rate range from 6e-4→6e-5 to 3e-4→3e-5 and warmup from 500 to 440 iterations.
  • Trained with BF16 on 2× RTX 4090 instead of 1× RTX 5090.
  • Introduced sampled training validation and a seven-task English zero-shot benchmark suite.

Full record: training-history/r002.md

r001 — experimental predecessor

  • Initial experimental release with a 12/12/768 architecture, 2,048-token context, and 95.96M nanoGPT-reported parameters.
  • Trained for four dataset passes and selected by deterministic full-validation reevaluation of retained checkpoints.
  • No downstream benchmark suite was run.

Full record and generation samples: training-history/r001.md