| --- |
| license: mit |
| --- |
| |
| # lfm2-quantum-128m (base checkpoint, step 2162) |
|
|
| LFM2-style hybrid quantum GPT, budget tier, trained with |
| [Quantum-GPT](https://github.com) `runs/run_lfm2_quantum_mini.sh`. |
|
|
| This is a **base** (pretrained, not instruction-tuned) checkpoint at step 2162 |
| (final step of pretraining) -- it does raw text continuation, not chat. |
|
|
| ## Architecture |
|
|
| - 16 layers, hybrid mixer: 10 gated short-conv blocks + 6 GQA attention blocks |
| (conv-first, pattern `CCCCCCCCCCAAAAAA`) |
| - `n_embd=1024`, 16 query heads / 8 KV heads (GQA), `head_dim=64`, `seq_len=1024` |
| - `vocab_size=32768` |
| - Quantum feed-forward network in every block (4 qubits, circuit depth 2, |
| simulated exactly in PyTorch -- see `nanochat/gpt.py`'s `QuantumMLP`) |
| - RoPE base `theta=1e6`, full (non-windowed) attention (`window_pattern=L`) |
| - Value embeddings disabled |
|
|
| Full config: `config.json` (metadata only) and `meta_002162.json` (exact |
| training config this checkpoint was produced with). |
|
|
| This is a **custom architecture**, not a `transformers` model -- there is no |
| `AutoModel` support. `config.json` is provided for discoverability/metadata |
| only; to actually load the model, use the bundled `nanochat/` package as |
| shown below. |
|
|
| ## Inference |
|
|
| This repo bundles the minimal `nanochat/` source needed to load and run the |
| model, so it's self-contained (no need to clone the full training repo). |
|
|
| ```bash |
| pip install torch tiktoken rustbpe filelock kernels |
| # download this repo, e.g.: |
| hf download MarkChenX/lfm2-quantum-128m --local-dir ./lfm2-quantum-128m |
| cd lfm2-quantum-128m |
| python inference.py --prompt "The history of quantum computing" |
| ``` |
|
|
| See `inference.py` for the full loading + generation code (~15 lines): it |
| calls `nanochat.checkpoint_manager.build_model(".", step=2162, ...)` to build |
| the model from `model_002162.pt` + `meta_002162.json`, loads the tokenizer |
| from `tokenizer/`, and streams tokens via the model's own `.generate()`. |
|
|
| ## Resuming pretraining |
|
|
| `optim_002162_rank0.pt` is the matching Muon/AdamW optimizer state (momentum |
| buffers etc.) for this step. To continue pretraining with the original |
| [Quantum-GPT](https://github.com) repo, place `model_002162.pt`, |
| `optim_002162_rank0.pt` and `meta_002162.json` under |
| `$NANOCHAT_BASE_DIR/base_checkpoints/lfm2-quantum-128m/`, then run: |
|
|
| ```bash |
| RESUME_FROM_STEP=2162 MODEL_TAG=lfm2-quantum-128m bash runs/run_lfm2_quantum_mini.sh |
| ``` |
|
|
| For SFT/RL instead of continued pretraining, only `model_002162.pt` + |
| `meta_002162.json` + `tokenizer/` are needed (the optimizer shard is |
| pretraining-only). |
|
|
