| --- |
| license: mit |
| language: [en] |
| tags: [text-generation, quantization, bitnet, tiny, from-scratch] |
| pipeline_tag: text-generation |
| --- |
| |
| # Artificial Stupidity — the tiny models (AS-0 … AS-5) |
|
|
| Six character-level language models **built from scratch** — no GPT-2, no |
| pretrained anything — that are identical in every respect except **how many |
| bits each weight is allowed**. The smallest is **83 KB**. |
|
|
| They are the control experiment for |
| [ayushmaninbox/artificial-stupidity](https://huggingface.co/ayushmaninbox/artificial-stupidity), |
| which is fluent because OpenAI paid for the fluency. These are not fluent, and |
| every parameter is homegrown. |
|
|
| ## The six |
|
|
| | | Weight precision | Params | Packed size | Val loss | |
| |---|---|--:|--:|--:| |
| | **AS-0** | 32-bit (control) | 815,488 | 3.1 MB | 1.642 | |
| | **AS-1** | 8-bit | 819,072 | 835 KB | — | |
| | **AS-2** | 4-bit | 819,072 | 448 KB | — | |
| | **AS-3** | 1.58-bit `{-1,0,+1}` | 819,072 | 216 KB | — | |
| | **AS-4** | 1-bit `{-1,+1}` | 819,072 | **169 KB** | 1.756 | |
| | **AS-5** | 1-bit + 8-bit activations | 350,784 | **83 KB** | — | |
|
|
| ## What they actually produce |
|
|
| Real output, same prompt, same seed (`python compare.py --seed 1337`): |
|
|
| **`why is the sky blue`** |
|
|
| | Model | Size | Answer | |
| |---|--:|---| |
| | AS-0 | 3.1 MB | *ask ays what is 17* | |
| | AS-1 | 835 KB | *no it isnt* | |
| | AS-2 | 448 KB | *a lot* | |
| | AS-3 | 216 KB | *ok* | |
| | AS-4 | 169 KB | *no it is rain* | |
| | AS-5 | 83 KB | *no 11 whats 4* | |
|
|
| **`what is 2 + 2`** — every one of them answers **`127`** (AS-0 says `1`). |
| They learned the *shape* of an arithmetic reply from the synthetic corpus |
| without learning arithmetic. |
|
|
| **`what is your name`** |
|
|
| | Model | Answer | |
| |---|---| |
| | AS-0 | *definitely not* | |
| | AS-2 | *definitely not ryand* | |
| | AS-4 | *dog* | |
| | AS-5 | *do 1542 plus 40 13* | |
|
|
| They hold a conversational **register** — correctly spelled, real chat moves — |
| while being completely disconnected from the question. AS-5 at 83 KB is where |
| it visibly breaks: it keeps the rhythm of internet chat but can no longer stay |
| on one topic for a whole sentence. |
|
|
| ## Quantization-aware, not post-hoc |
|
|
| You **cannot** train a normal model and round its weights to 1 bit afterwards — |
| you get static, and worse, you cannot tell "compression worked" from "my code is |
| broken". These models know they are being squashed *while they learn*, via a |
| straight-through estimator, so they route around the damage. |
|
|
| 1 bit is not a 32× saving but about **20×**: embeddings, LayerNorms and one fp16 |
| scale per weight row stay high-precision. BitNet keeps them too. |
|
|
| ## Limitations |
|
|
| - **Character-level**, so they cannot spell reliably — "mitochondria" is twelve |
| consecutive guesses and they lose that bet. |
| - Trained on 9.8 MB of Twitch chat, Reddit, YouTube transcripts and lyrics. |
| - They are **not** useful. They are a measurement of where language breaks. |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/ayushmaninbox/artificial-stupidity |
| cd artificial-stupidity/as-text-model |
| pip install -r requirements.txt |
| python generate.py AS-4 --prompt "hello" |
| python compare.py # all six, side by side |
| ``` |
|
|
| ## License |
|
|
| MIT for the code and weights. The training corpus is scraped third-party text — |
| see the [dataset card](https://huggingface.co/datasets/ayushmaninbox/artificial-stupidity-corpus). |
|
|
