QuantaSparkLabs commited on
Commit
fb76558
Β·
verified Β·
1 Parent(s): 1fd002d

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +232 -0
README.md ADDED
@@ -0,0 +1,232 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ language:
4
+ - en
5
+ pipeline_tag: text-generation
6
+ tags:
7
+ - transformer
8
+ - gpt2
9
+ - tiny
10
+ - atom
11
+ - experimental
12
+ - humor
13
+ - whirlwindai
14
+ ---
15
+
16
+ <div align="center">
17
+
18
+ <img src="https://readme-typing-svg.demolab.com?font=Space+Grotesk&weight=700&size=27&duration=1900&pause=900&color=22C55E&center=true&vCenter=true&width=850&lines=AtomZephyr;27+Parameters.;Almost+an+LLM.;Mostly+a+Science+Experiment.;Powered+by+Pure+Curiosity." />
19
+
20
+ <br>
21
+
22
+ <img src="https://img.shields.io/badge/Parameters-27-22C55E?style=for-the-badge">
23
+ <img src="https://img.shields.io/badge/Architecture-GPT--2-10B981?style=for-the-badge">
24
+ <img src="https://img.shields.io/badge/Status-Experimental-14B8A6?style=for-the-badge">
25
+ <img src="https://img.shields.io/badge/Braincells-27-06B6D4?style=for-the-badge">
26
+
27
+ <br><br>
28
+
29
+ <img src="https://capsule-render.vercel.app/api?type=soft&height=190&text=AtomZephyr&fontColor=ffffff&fontSize=48&animation=blinking&color=0:22C55E,100:06B6D4"/>
30
+
31
+ </div>
32
+
33
+ ---
34
+
35
+ # The Idea
36
+
37
+ <div align="center">
38
+
39
+ <table width="92%">
40
+
41
+ <tr>
42
+
43
+ <td align="center">
44
+
45
+ ## What if a transformer became... microscopic?
46
+
47
+ AtomZephyr explores one of the smallest practical transformer architectures ever built.
48
+
49
+ Not because anyone asked for it.
50
+
51
+ Because someone eventually had to answer the question:
52
+
53
+ **"How absurdly small can an AI become before it forgets how to AI?"**
54
+
55
+ Turns out...
56
+
57
+ **27 parameters is still technically enough.**
58
+
59
+ </td>
60
+
61
+ </tr>
62
+
63
+ </table>
64
+
65
+ </div>
66
+
67
+ ---
68
+
69
+ # Why?
70
+
71
+ Most AI models compete by getting bigger.
72
+
73
+ AtomZephyr competes by removing parameters until people start questioning whether it's still a neural network.
74
+
75
+ Every parameter had to earn its place.
76
+
77
+ Most didn't.
78
+
79
+ ---
80
+
81
+ # Specifications
82
+
83
+ | Property | Value |
84
+ |-----------|-------|
85
+ | Parameters | **27** |
86
+ | Architecture | GPT-2 |
87
+ | Layers | 1 |
88
+ | Attention Heads | 1 |
89
+ | Embedding Size | 1 |
90
+ | FFN Size | 1 |
91
+ | Context Length | 4 |
92
+ | Vocabulary | 5 Tokens |
93
+ | Model Size | <5 KB |
94
+ | Training Time | ~6 Seconds (CPU) |
95
+
96
+ ---
97
+
98
+ # Performance
99
+
100
+ | Test | Result |
101
+ |------|--------|
102
+ | Understand English | ❌ |
103
+ | Write Code | ❌ |
104
+ | Solve Math | ❌ |
105
+ | Generate "abba" | βœ… |
106
+ | Break Expectations | βœ… |
107
+
108
+ ---
109
+
110
+ # Quick Start
111
+
112
+ ```python
113
+ from transformers import AutoTokenizer, AutoModelForCausalLM
114
+
115
+ tokenizer = AutoTokenizer.from_pretrained("WhirlwindAI/AtomZephyr")
116
+ model = AutoModelForCausalLM.from_pretrained("WhirlwindAI/AtomZephyr")
117
+
118
+ prompt = "a"
119
+
120
+ inputs = tokenizer(prompt, return_tensors="pt")
121
+
122
+ outputs = model.generate(
123
+ **inputs,
124
+ do_sample=True,
125
+ temperature=1.7,
126
+ max_length=4
127
+ )
128
+
129
+ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
130
+ ```
131
+
132
+ Possible output
133
+
134
+ ```
135
+ abaa
136
+ ```
137
+
138
+ Groundbreaking.
139
+
140
+ ---
141
+
142
+ # Example Conversation
143
+
144
+ **User**
145
+
146
+ > Tell me a joke.
147
+
148
+ **AtomZephyr**
149
+
150
+ ```
151
+ abba
152
+ ```
153
+
154
+ Technically...
155
+
156
+ that's an answer.
157
+
158
+ ---
159
+
160
+ # Scientific Achievement
161
+
162
+ Removing parameters is easy.
163
+
164
+ Keeping a transformer alive afterwards...
165
+
166
+ isn't.
167
+
168
+ AtomZephyr exists purely to explore the absolute lower limits of transformer architectures while remaining a real, trainable language model.
169
+
170
+ Whether it's useful is a completely different discussion.
171
+
172
+ ---
173
+
174
+ # Awards
175
+
176
+ πŸ₯‡ Smallest Model That Still Has Self-Respect
177
+
178
+ πŸ† Best Binary Poetry Generator
179
+
180
+ πŸ₯ˆ Most Efficient Waste Of Six Seconds
181
+
182
+ πŸŽ–οΈ Official Representative Of Tiny AI
183
+
184
+ ---
185
+
186
+ # Limitations
187
+
188
+ AtomZephyr should **not** be used for:
189
+
190
+ - Programming
191
+ - Translation
192
+ - Question Answering
193
+ - Homework
194
+ - Anything important
195
+
196
+ It performs significantly better when asked to do absolutely nothing useful.
197
+
198
+ ---
199
+
200
+ # Fun Facts
201
+
202
+ - Fits inside most PNG images.
203
+ - Smaller than many neural network tutorials.
204
+ - Downloads faster than this README loads.
205
+ - Has fewer parameters than some calculator manuals have pages.
206
+
207
+ ---
208
+
209
+ # License
210
+
211
+ MIT
212
+
213
+ Take it apart.
214
+
215
+ Make it smaller.
216
+
217
+ Break another record.
218
+
219
+ ---
220
+
221
+ <div align="center">
222
+
223
+ ### Built by WhirlwindAI
224
+
225
+ *"Sometimes progress isn't measured in billions... it's measured in what you can remove."*
226
+
227
+ <br>
228
+
229
+ <img src="https://capsule-render.vercel.app/api?type=waving&height=120&section=footer&color=0:22C55E,100:06B6D4"/>
230
+
231
+ </div>
232
+