File size: 2,539 Bytes
b4207ce 68a7bb4 d605bf1 68a7bb4 d605bf1 68a7bb4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | ---
license: mit
---
# lfm2-quantum-128m (base checkpoint, step 2162)
LFM2-style hybrid quantum GPT, budget tier, trained with
[Quantum-GPT](https://github.com) `runs/run_lfm2_quantum_mini.sh`.
This is a **base** (pretrained, not instruction-tuned) checkpoint at step 2162
(final step of pretraining) -- it does raw text continuation, not chat.
## Architecture
- 16 layers, hybrid mixer: 10 gated short-conv blocks + 6 GQA attention blocks
(conv-first, pattern `CCCCCCCCCCAAAAAA`)
- `n_embd=1024`, 16 query heads / 8 KV heads (GQA), `head_dim=64`, `seq_len=1024`
- `vocab_size=32768`
- Quantum feed-forward network in every block (4 qubits, circuit depth 2,
simulated exactly in PyTorch -- see `nanochat/gpt.py`'s `QuantumMLP`)
- RoPE base `theta=1e6`, full (non-windowed) attention (`window_pattern=L`)
- Value embeddings disabled
Full config: `config.json` (metadata only) and `meta_002162.json` (exact
training config this checkpoint was produced with).
This is a **custom architecture**, not a `transformers` model -- there is no
`AutoModel` support. `config.json` is provided for discoverability/metadata
only; to actually load the model, use the bundled `nanochat/` package as
shown below.
## Inference
This repo bundles the minimal `nanochat/` source needed to load and run the
model, so it's self-contained (no need to clone the full training repo).
```bash
pip install torch tiktoken rustbpe filelock kernels
# download this repo, e.g.:
hf download MarkChenX/lfm2-quantum-128m --local-dir ./lfm2-quantum-128m
cd lfm2-quantum-128m
python inference.py --prompt "The history of quantum computing"
```
See `inference.py` for the full loading + generation code (~15 lines): it
calls `nanochat.checkpoint_manager.build_model(".", step=2162, ...)` to build
the model from `model_002162.pt` + `meta_002162.json`, loads the tokenizer
from `tokenizer/`, and streams tokens via the model's own `.generate()`.
## Resuming pretraining
`optim_002162_rank0.pt` is the matching Muon/AdamW optimizer state (momentum
buffers etc.) for this step. To continue pretraining with the original
[Quantum-GPT](https://github.com) repo, place `model_002162.pt`,
`optim_002162_rank0.pt` and `meta_002162.json` under
`$NANOCHAT_BASE_DIR/base_checkpoints/lfm2-quantum-128m/`, then run:
```bash
RESUME_FROM_STEP=2162 MODEL_TAG=lfm2-quantum-128m bash runs/run_lfm2_quantum_mini.sh
```
For SFT/RL instead of continued pretraining, only `model_002162.pt` +
`meta_002162.json` + `tokenizer/` are needed (the optimizer shard is
pretraining-only).
|