README.md

---
license: apache-2.0
language: [en]
tags: [udgamlabs, small-language-model, trained-from-scratch, research-release, checkpoints, training-dynamics]
---

![Laghu-Checkpoints · Udgam Labs](./banner.png)

**UDGAM LABS** · Training checkpoints · 152M parameters, every surviving state

# Laghu-Checkpoints

The raw training states behind [Laghu](https://huggingface.co/udgamlabs/Laghu), a 152M-parameter chat model trained from
scratch for $146 of pretraining compute: model weights, EMA weights and optimiser states from the end of pretraining
and of the 32K stretch, all three supervised fine-tuning runs (1, 2 and 3 passes), the reinforcement-learning run that
failed its gate, and the run logs. Together with the code on GitHub, they let you resume training, re-run the
fine-tuning, or study how the model changed between stages.

**Try it:** [the chat model](https://huggingface.co/udgamlabs/Laghu) · [code on GitHub](https://github.com/udgamlabs/laghu) · [udgamlabs.com/models/laghu](https://udgamlabs.com/models/laghu)

## Quick start

```python
import torch
from huggingface_hub import hf_hub_download

p = hf_hub_download("udgamlabs/Laghu-Checkpoints", "pretrain/step_076294/model_ema_076294.pt")
state = torch.load(p, map_location="cpu", weights_only=True)   # a state dict for nanochat's GPT model
```

These are nanochat files (`torch.save`), not transformers checkpoints. Load them with the training code in
[udgamlabs/laghu](https://github.com/udgamlabs/laghu) (`nanochat/`), or convert one to the transformers format with
`python -m scripts.export_hf --checkpoint-dir <folder> --weights final --out <dir>`. Ready-to-use transformers weights
are in [Laghu](https://huggingface.co/udgamlabs/Laghu), [Laghu-Base](https://huggingface.co/udgamlabs/Laghu-Base) and
[Laghu-Base-32K](https://huggingface.co/udgamlabs/Laghu-Base-32K), and in the `hf/` folders below.

## What is here

| Folder | Stage | Files |
|---|---|---|
| `pretrain/step_076000/` | pretraining, 99.6% | `model_076000.pt` (weights), `ema_076000.pt` (EMA), `optim_076000_rank{0-7}.pt` (one optimiser shard per GPU of the 8-GPU run), `meta_076000.json` |
| `pretrain/step_076294/` | end of pretraining | as above, plus `model_ema_076294.pt` (the EMA weights as a state dict; this is Laghu-Base) |
| `pretrain/logs/`, `pretrain/summary.txt` | | the main run's log (every step's loss, validation every 2,000 steps) |
| `ctx32k/step_001750/`, `ctx32k/step_001900/` | 32K stretch | `model_*.pt`, `optim_*_rank0.pt`, `meta_*.json`; step 1,900 is Laghu-Base-32K and the start of SFT |
| `ctx32k/logs/`, `ctx32k/eval/` | | training log; recall and short-text checks |
| `sft/1-pass/`, `sft/2-passes/`, `sft/3-passes/` | supervised fine-tuning | `ckpt/` (weights and optimiser at the halfway and final steps), `hf/` (transformers export of the final step), `logs/`; 2 passes is the released Laghu |
| `rl-r1-failed/` | reinforcement learning (GRPO + distillation from Gargi), 40 steps | `hf/` export, `eval/`, `logs/`; it failed its pre-registered gate (dev GSM8K 8.6% → 8.2%, bar 11.6%) and is published for study only |

File formats:
- `model_*.pt` and `model_ema_*.pt`: state dicts (the EMA ones in fp32), loadable with `torch.load(..., weights_only=True)`.
- `ema_*.pt`: the EMA as a list of fp32 tensors in `model.parameters()` order, used to resume the EMA.
- `optim_*.pt`: optimiser state dicts (Muon for matrices, Adam for embeddings, output and scalars), loadable with
  `weights_only=True`.
- `meta_*.json`: model config, run settings, data-loader position and validation scores at that step. Measure a
  checkpoint yourself rather than reading scores from its meta file.

## Results

Measured on the exported weights (details on the model cards and in the
[technical report](https://udgamlabs.com/research/laghu)):

| Checkpoint | Bits per byte on new text (lower is better) | Notes |
|---|---|---|
| pretraining step 76,294, EMA (Laghu-Base) | 0.919 | SmolLM2-135M 0.941, Falcon-H1-Tiny-90M-Base 0.979 |
| 32K step 1,900 (Laghu-Base-32K) | 0.921 | hidden-code recall 96-100% up to 16K tokens |
| SFT 1 / 2 / 3 passes | held-out SFT loss 1.724 / 1.784 / 2.036 | 2 passes was picked by a generation-based rule (60.0% judge win rate over 1 pass) |

## Where it falls short

- **Most of the pretraining trajectory is missing.** Checkpoints were saved every 2,000 steps, but the uploader kept
  only the newest two on the main branch, and a storage clean-up on 2026-10-21 deleted the older versions (steps
  2,000-74,000) and about six earlier 32K checkpoints. That was our mistake. Only the late states here survive, so this
  repository cannot show how Laghu learned over the whole run.
- The 32K stretch states have one optimiser shard (one GPU); pretraining states have eight. Resuming on a different GPU
  count needs re-sharding.
- The RL run is a failed experiment, not an improvement.

## How it was trained

Pretraining: 76,294 steps × 524,288 tokens (40.0B token positions) at 2,048 context on 8× A100 80GB, 13.2 hours, $146.26;
Muon; token bags for the first 25% of steps; cool-down data from 80%; EMA from 90% (β 0.998). 32K stretch: 1,900 steps at
32,768 context on one RTX PRO 6000. SFT: 41,467 conversations, sequence 16,384, batch 262,144 tokens. Every setting and
script: [GitHub](https://github.com/udgamlabs/laghu).

## Intended use and limitations

Research on training dynamics, continued pretraining, fine-tuning and reproduction. The weights carry no safety tuning
beyond light SFT data.

## Licence

Apache-2.0. Copyright 2026 Udgam Labs LLP. Training-data credits:
[DATA_LICENSES](https://github.com/udgamlabs/laghu/blob/main/docs/DATA_LICENSES.md).

## Citation

```bibtex
@misc{udgamlabs2026laghu,
  title  = {Laghu: a fully open 152M chat model trained from scratch},
  author = {Prasad, Jitesh K},
  year   = {2026},
  howpublished = {Udgam Labs},
  url    = {https://huggingface.co/udgamlabs/Laghu}
}
```

---

**About Udgam Labs.** Udgam Labs is an independent AI research lab. We work from first principles: before we build, we understand why things work, whether we are training a model, fine-tuning one or applying AI to a new problem. Udgam (उद्गम) is Sanskrit for origin. [udgamlabs.com](https://udgamlabs.com)