QuantaSparkLabs commited on
Commit
4cd0f07
Β·
verified Β·
1 Parent(s): bb329f2

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +246 -0
README.md ADDED
@@ -0,0 +1,246 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ language:
4
+ - en
5
+ pipeline_tag: text-generation
6
+ tags:
7
+ - gpt2
8
+ - tiny
9
+ - transformer
10
+ - experimental
11
+ - humor
12
+ - text-generation
13
+ - whirlwindai
14
+ ---
15
+
16
+ <div align="center">
17
+
18
+ <img src="https://readme-typing-svg.demolab.com?font=Space+Grotesk&weight=700&size=26&duration=1800&pause=900&color=A855F7&center=true&vCenter=true&width=820&lines=TinyZephyr;1%2C272+Parameters.;Technically+an+LLM.;Powered+by+Questionable+Engineering.;Runs+Before+You+Finish+Blinking." />
19
+
20
+ <br>
21
+
22
+ <img src="https://img.shields.io/badge/Parameters-1,272-8B5CF6?style=for-the-badge">
23
+ <img src="https://img.shields.io/badge/Architecture-GPT--2-A855F7?style=for-the-badge">
24
+ <img src="https://img.shields.io/badge/Status-Experimental-7C3AED?style=for-the-badge">
25
+ <img src="https://img.shields.io/badge/Humor-Included-6366F1?style=for-the-badge">
26
+
27
+ <br><br>
28
+
29
+ <img src="https://capsule-render.vercel.app/api?type=rounded&height=190&text=TinyZephyr&fontColor=ffffff&fontSize=48&animation=twinkling&color=0:8B5CF6,50:A855F7,100:2563EB"/>
30
+
31
+ </div>
32
+
33
+ ---
34
+
35
+ # The Idea
36
+
37
+ <div align="center">
38
+
39
+ <table width="92%">
40
+ <tr>
41
+ <td align="center">
42
+
43
+ ## Bigger isn't always better.
44
+
45
+ TinyZephyr explores the opposite extreme.
46
+
47
+ Instead of billions of parameters, this model asks one very important question:
48
+
49
+ **How ridiculously small can a transformer become before it completely loses its mind?**
50
+
51
+ The answer is... surprisingly entertaining.
52
+
53
+ </td>
54
+ </tr>
55
+ </table>
56
+
57
+ </div>
58
+
59
+ ---
60
+
61
+ # Why?
62
+
63
+ Nobody needed this.
64
+
65
+ Nobody requested it.
66
+
67
+ Nobody funded it.
68
+
69
+ Yet somehow...
70
+
71
+ **TinyZephyr exists.**
72
+
73
+ It was built purely as an experiment to explore the lower limits of transformer architectures while proving that even microscopic language models deserve beautiful documentation.
74
+
75
+ ---
76
+
77
+ # Specifications
78
+
79
+ | Property | Value |
80
+ |-----------|-------|
81
+ | Parameters | **1,272** |
82
+ | Architecture | GPT-2 |
83
+ | Layers | 1 |
84
+ | Attention Heads | 1 |
85
+ | Embedding Size | 8 |
86
+ | Context Length | 32 |
87
+ | Vocabulary | 50 Tokens |
88
+ | Model Size | ~25 KB |
89
+ | Training Time | ~4 Minutes (CPU) |
90
+
91
+ ---
92
+
93
+ # Benchmark
94
+
95
+ | Task | Result |
96
+ |------|--------|
97
+ | Write Python | ❌ |
98
+ | Solve Math | ❌ |
99
+ | Explain Physics | ❌ |
100
+ | Generate Gibberish | βœ… |
101
+ | Exist | βœ… |
102
+
103
+ ---
104
+
105
+ # Quick Start
106
+
107
+ ```python
108
+ from transformers import AutoTokenizer, AutoModelForCausalLM
109
+
110
+ tokenizer = AutoTokenizer.from_pretrained("WhirlwindAI/TinyZephyr")
111
+ model = AutoModelForCausalLM.from_pretrained("WhirlwindAI/TinyZephyr")
112
+
113
+ prompt = "The meaning of life is"
114
+
115
+ inputs = tokenizer(prompt, return_tensors="pt")
116
+
117
+ outputs = model.generate(
118
+ **inputs,
119
+ do_sample=True,
120
+ temperature=1.6,
121
+ max_length=32
122
+ )
123
+
124
+ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
125
+ ````
126
+
127
+ Possible output:
128
+
129
+ ```
130
+ The meaning of life is...
131
+ xqw fjczqnv lpoqa yv
132
+ ```
133
+
134
+ Beautiful.
135
+
136
+ ---
137
+
138
+ # Example Conversation
139
+
140
+ **You**
141
+
142
+ > Write a poem about space.
143
+
144
+ **TinyZephyr**
145
+
146
+ > moon potato quantum fish sandwich
147
+
148
+ Mission accomplished.
149
+
150
+ ---
151
+
152
+ # System Requirements
153
+
154
+ CPU
155
+
156
+ > Yes.
157
+
158
+ GPU
159
+
160
+ > Optional.
161
+
162
+ RAM
163
+
164
+ > If your browser opens, you're probably fine.
165
+
166
+ Storage
167
+
168
+ > Less than most PNG files.
169
+
170
+ ---
171
+
172
+ # Frequently Asked Questions
173
+
174
+ ### Is this useful?
175
+
176
+ Not particularly.
177
+
178
+ ### Is this serious research?
179
+
180
+ Surprisingly... yes.
181
+
182
+ ### Can it replace ChatGPT?
183
+
184
+ Only if your expectations are extremely flexible.
185
+
186
+ ### Why did you build this?
187
+
188
+ Curiosity.
189
+
190
+ And because somebody had to.
191
+
192
+ ---
193
+
194
+ # Awards
195
+
196
+ πŸ† Fastest Model To Finish Inference
197
+
198
+ πŸ₯‡ Most Parameters Removed Without Deleting Everything
199
+
200
+ πŸ₯ˆ Best Random Sentence Generator
201
+
202
+ πŸ₯‰ Self-Proclaimed Champion of Tiny AI
203
+
204
+ ---
205
+
206
+ # Limitations
207
+
208
+ TinyZephyr was never trained to be helpful.
209
+
210
+ It doesn't know facts.
211
+
212
+ It doesn't reason.
213
+
214
+ It doesn't write code.
215
+
216
+ It mostly produces beautifully random nonsense.
217
+
218
+ And that's exactly what it was designed to do.
219
+
220
+ ---
221
+
222
+ # License
223
+
224
+ MIT
225
+
226
+ Use it.
227
+
228
+ Benchmark it.
229
+
230
+ Laugh at it.
231
+
232
+ Make it even smaller.
233
+
234
+ ---
235
+
236
+ <div align="center">
237
+
238
+ ### Built by WhirlwindAI
239
+
240
+ *Sometimes the best experiments begin with terrible ideas.*
241
+
242
+ <br>
243
+
244
+ <img src="https://capsule-render.vercel.app/api?type=waving&height=120&section=footer&color=0:8B5CF6,100:2563EB"/>
245
+
246
+ </div>