dharun2049 commited on
Commit
4548d19
·
verified ·
1 Parent(s): 212e2fd

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -90
README.md CHANGED
@@ -42,93 +42,3 @@ Input -> [Embed] -> [Block x 4] -> [RMSNorm] -> [LM Head] -> Output
42
  | Flash Attention 2 | Stanford | O(1) memory fused attention kernel |
43
  | torch.compile | PyTorch 2.0+ | Graph compilation with operator fusion |
44
 
45
- ## Specs
46
-
47
- | Parameter | Value |
48
- |-----------|-------|
49
- | Layers | 4 |
50
- | Embed dim | 256 |
51
- | Heads | 4 |
52
- | MLA latent | 32 |
53
- | FFN hidden | 682 |
54
- | Context | 256 |
55
- | Sliding window | 128 |
56
- | Vocab | 50,257 (GPT-2 BPE) |
57
- | Params | 15,586,816 |
58
- | Checkpoint | ~31 MB (FP16) |
59
-
60
- ## Files
61
-
62
- ```
63
- train_goat_gpt_nano.py Training script (auto-downloads dataset)
64
- app.py HF Spaces deployment (Gradio UI)
65
- requirements.txt Dependencies
66
- goat_gpt_nano.ht Trained model output (generated)
67
- ```
68
-
69
- ## Build / Run
70
-
71
- ### Train
72
-
73
- ```bash
74
- pip install -r requirements.txt
75
- python train_goat_gpt_nano.py
76
- ```
77
-
78
- Outputs:
79
- - `goat_gpt_nano.ht` -- deployable model
80
- - `goat_gpt_nano_gen.pt` -- generation checkpoint
81
- - `best.pt` -- best validation checkpoint
82
- - `ckpt_*.pt` -- intermediate checkpoints
83
-
84
- ### Deploy (Hugging Face Spaces)
85
-
86
- 1. Create Space -> Gradio SDK
87
- 2. Upload:
88
- - `goat_gpt_nano.ht`
89
- - `app.py`
90
- - `requirements.txt`
91
-
92
- ## Training Config
93
-
94
- | Parameter | Value |
95
- |-----------|-------|
96
- | Batch size | 16 |
97
- | Grad accum | 2 (effective = 32) |
98
- | Max steps | 5000 |
99
- | Warmup | 250 |
100
- | Stable | 3000 |
101
- | Decay | 1750 |
102
- | LR | 8e-4 -> 8e-5 |
103
- | Optimizer | Lion |
104
- | Precision | FP16 |
105
- | Compile | default (T4-safe) |
106
-
107
- ## Dataset
108
-
109
- - Source: WikiText-2 (auto-downloaded from HuggingFace)
110
- - Fallback chain: wikitext -> wikitext-2-raw-v1 -> Salesforce/wikitext -> local file
111
- - Train tokens: ~2.4M
112
- - Val tokens: ~250K
113
-
114
- ## GPU
115
-
116
- - Recommended: T4 (Google Colab)
117
- - Cost: ~5-10 compute units per run
118
- - Time: ~3-5 hours
119
- - VRAM: <2 GB
120
-
121
- ## Expected Results
122
-
123
- | Metric | Value |
124
- |--------|-------|
125
- | Val loss | ~3.5-4.7 |
126
- | Val PPL | ~33-110 |
127
- | Throughput | ~50-80K tok/s |
128
-
129
- ## Notes
130
-
131
- - `TORCHINDUCTOR_MAX_AUTOTUNE_GEMM=0` set before torch import (T4 compatibility)
132
- - `torch.amp.GradScaler('cuda')` and `torch.amp.autocast('cuda')` used (PyTorch 2.6+)
133
- - `trust_remote_code` removed (datasets library deprecated)
134
- - Warning suppression active for clean logs
 
42
  | Flash Attention 2 | Stanford | O(1) memory fused attention kernel |
43
  | torch.compile | PyTorch 2.0+ | Graph compilation with operator fusion |
44