P1yansh commited on
Commit
767bb48
·
1 Parent(s): b5b304d

Update README tone to third person and remove emojis

Browse files
Files changed (1) hide show
  1. README.md +20 -21
README.md CHANGED
@@ -1,32 +1,31 @@
1
- # 🍼 Nano-GLM (GLM-5.2 Baby 120M) - From Scratch
2
 
3
- This repository contains a from-scratch implementation and pretraining script for a "baby" version (~120M parameters) of **GLM-5.2** (GLM MoE DSA). It is heavily inspired by Andrej Karpathy's `nanoGPT` and aims to be highly educational.
4
 
5
- The model is small enough to train on a single consumer laptop GPU (e.g., RTX 4050 6GB VRAM) but includes all the cutting-edge architectural innovations of modern frontier models.
6
 
7
- ## Architectural Features Implemented
8
 
9
- This isn't just a standard Transformer. It implements three major innovations from recent frontier models (like DeepSeek-V3 and GLM-5):
10
 
11
- 1. **MLA (Multi-Latent Attention):** Compresses the attention mechanism using LoRA-style projections to drastically save VRAM during training and inference.
12
  2. **DSA (DeepSeek Sparse Attention):** Selects only the most relevant tokens to attend to via a learned indexer, rather than attending to the entire context uniformly.
13
- 3. **MoE (Mixture of Experts):** Uses a fine-grained sigmoid-routed mixture of experts alongside a shared expert, activating only a subset of parameters per token.
14
 
15
- ## 🚀 Training Features
16
 
17
- The training loop (`train_glm5.py`) is highly optimized for limited hardware (6GB VRAM) while maximizing throughput (~4,900 tokens/sec on an RTX 4050):
18
- - **Mixed Precision:** Uses `bfloat16` and TF32 Tensor Cores.
19
- - **Gradient Checkpointing:** Recomputes forward passes during backprop to save ~40% VRAM.
20
- - **Gradient Accumulation:** Achieves large effective batch sizes on a single GPU.
21
- - **WSD (Warmup-Stable-Decay) Learning Rate Schedule:**
22
- Supports multi-phase training by holding the learning rate at peak for a "stable" exploration phase before initiating a steep cosine decay. (Controlled via `--stable_iters`).
23
 
24
- ## 📚 Educational Guide
25
 
26
- If you are new to LLM pretraining, learning rates, loss curves, and scaling laws, check out the included beginner guide:
27
- 👉 **[LLM Training Guide for Beginners](llm_training_guide.md)**
28
 
29
- ## 🛠️ Usage
30
 
31
  ### Installation
32
  ```bash
@@ -34,7 +33,7 @@ pip install -r requirements.txt
34
  ```
35
 
36
  ### Training
37
- To train the model on a single GPU with the WSD schedule (holding LR stable for 217,000 steps):
38
 
39
  ```bash
40
  python train_glm5.py \
@@ -51,11 +50,11 @@ python train_glm5.py \
51
  ```
52
 
53
  ### Generation / Sampling
54
- To sample text from your best trained checkpoint:
55
 
56
  ```bash
57
  python train_glm5.py --eval_only --ckpt out_glm5/ckpt_best.pt --prompt "The future of AI is"
58
  ```
59
 
60
- ## ⚖️ License
61
  MIT License
 
1
+ # Nano-GLM (GLM-5.2 Baby 120M) - From Scratch
2
 
3
+ This repository contains a from-scratch implementation and pretraining script for a baby version (~120M parameters) of GLM-5.2 (GLM MoE DSA). The project is heavily inspired by Andrej Karpathy's nanoGPT and aims to serve as a highly educational resource.
4
 
5
+ The model is designed to be small enough to train on a single consumer laptop GPU (e.g., RTX 4050 6GB VRAM) while incorporating cutting-edge architectural innovations found in modern frontier models.
6
 
7
+ ## Architectural Features Implemented
8
 
9
+ This implementation goes beyond a standard Transformer by incorporating three major innovations from recent frontier models (such as DeepSeek-V3 and GLM-5):
10
 
11
+ 1. **MLA (Multi-Latent Attention):** Compresses the attention mechanism using LoRA-style projections to drastically reduce VRAM usage during training and inference.
12
  2. **DSA (DeepSeek Sparse Attention):** Selects only the most relevant tokens to attend to via a learned indexer, rather than attending to the entire context uniformly.
13
+ 3. **MoE (Mixture of Experts):** Employs a fine-grained sigmoid-routed mixture of experts alongside a shared expert, activating only a subset of parameters per token.
14
 
15
+ ## Training Features
16
 
17
+ The training loop (train_glm5.py) is highly optimized for limited hardware (6GB VRAM) while maximizing throughput (achieving ~4,900 tokens/sec on an RTX 4050):
18
+ - **Mixed Precision:** Utilizes bfloat16 and TF32 Tensor Cores.
19
+ - **Gradient Checkpointing:** Recomputes forward passes during backpropagation to reduce VRAM consumption by approximately 40%.
20
+ - **Gradient Accumulation:** Enables large effective batch sizes on a single GPU.
21
+ - **WSD (Warmup-Stable-Decay) Learning Rate Schedule:** Supports multi-phase training by holding the learning rate at a peak for a stable exploration phase before initiating a steep cosine decay (controlled via the --stable_iters parameter).
 
22
 
23
+ ## Educational Guide
24
 
25
+ For individuals new to LLM pretraining, learning rates, loss curves, and scaling laws, an included beginner guide is available:
26
+ [LLM Training Guide for Beginners](llm_training_guide.md)
27
 
28
+ ## Usage
29
 
30
  ### Installation
31
  ```bash
 
33
  ```
34
 
35
  ### Training
36
+ To train the model on a single GPU using the WSD schedule (holding the learning rate stable for 217,000 steps), execute the following command:
37
 
38
  ```bash
39
  python train_glm5.py \
 
50
  ```
51
 
52
  ### Generation / Sampling
53
+ To sample text from the best trained checkpoint:
54
 
55
  ```bash
56
  python train_glm5.py --eval_only --ckpt out_glm5/ckpt_best.pt --prompt "The future of AI is"
57
  ```
58
 
59
+ ## License
60
  MIT License