Lumi-node commited on
Commit
1370bb3
·
verified ·
1 Parent(s): c898fd5

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +58 -0
README.md ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - model-garage
4
+ - decomposed
5
+ - gpt2
6
+ - interpretability
7
+ - model-surgery
8
+ license: apache-2.0
9
+ library_name: model-garage
10
+ ---
11
+
12
+ # GPT-2 Decomposed — Model Garage
13
+
14
+ Full component-level decomposition of GPT-2 (124M parameters) using [Model Garage](https://github.com/Lumi-node/model-garage).
15
+
16
+ ## What's Here
17
+
18
+ 64 individually extracted `nn.Module` components:
19
+
20
+ | Component Type | Count | Dimensions |
21
+ |---------------|-------|-----------|
22
+ | Attention (self_attn) | 12 | 768d, 12 heads, 64 head_dim |
23
+ | Feed-Forward (mlp) | 12 | 768→3072→768 |
24
+ | Layer Norm (ln_1, ln_2) | 24 | 768d |
25
+ | Full Layers | 12 | 768d |
26
+ | Embeddings | 2 | token (50257→768), position (1024→768) |
27
+ | Output Head | 1 | 768→50257 |
28
+ | Final Norm | 1 | 768d |
29
+
30
+ ## Usage
31
+
32
+ ```python
33
+ from model_garage.extract.pytorch import PyTorchExtractor
34
+
35
+ extractor = PyTorchExtractor("gpt2")
36
+ extractor.load_model()
37
+
38
+ # Extract any component
39
+ attn = extractor.extract_component("self_attention", layer_idx=6)
40
+ ffn = extractor.extract_component("feed_forward", layer_idx=6)
41
+
42
+ # Test in isolation
43
+ from model_garage.extract.pytorch import ComponentTester
44
+ tester = ComponentTester()
45
+ print(tester.test_attention(attn))
46
+ ```
47
+
48
+ ## Install
49
+
50
+ ```bash
51
+ pip install model-garage
52
+ ```
53
+
54
+ ## Links
55
+
56
+ - [Model Garage GitHub](https://github.com/Lumi-node/model-garage)
57
+ - [PyPI](https://pypi.org/project/model-garage/)
58
+ - [Research Papers](https://github.com/Lumi-node/model-garage/tree/main/research/papers)