GPT-Usenet-4
One of the largest LLMs possible to create in Google Colab. Trained using 8 GB of USENET posts along with 6 GB of miscellaneous BBS posts, digitized books, and text documents. Supervised fine-tuning should be performed before use.
Technical Information
| Layers | 36 |
| Heads | 20 |
| Embeddings | 1280 |
| Context Window | 32768 tokens |
| Tokenizer | GPT-2 BPE |