---
license: mit
language:
- en
tags:
- text-generation-inference
pipeline_tag: text-generation
---

![GPTUsenet2](https://cdn-uploads.huggingface.co/production/uploads/64b7618e2f5a966b972e9978/rl907PheZr8k0URxR-UVQ.png)

## GPT-Usenet-2
An 81-million parameter LLM using GPT-2 encodings.
Trained using 10GB of USENET posts along with over 1 GB of miscellaneous BBS posts, digitized books, and text documents.
Supervised fine-tuning should be performed before use.

## Purpose of GPT-Usenet-2
LLMs are all currently focused on becoming larger and larger, able to do more and more. However, this just makes them jack of all trades, master of none. GPT-Usenet takes a different approach. Instead of trying to do everything perfectly, GPT-Usenet offers a digital stem cell, which can then be finetuned into a single, specialized role and run in parallel with copies of itself.

## Technical Information
|                                 |     |
|---------------------------------|----:|
|Layers                           |10|
|Heads                            |10|
|Embeddings                       |640|
|Context Window                   |1024 tokens|
|Tokenizer                        |GPT-2 BPE|


## Training Information
|                                 |     |
|---------------------------------|----:|
|Training Loss                    |around 2.0254|
|Validation Loss                  |around 1.9795|
|Device                           |Google Colab L4, Google Colab A100|
|Training Time                    |16 Hours|


## Example Syntax

|                                 |     |
|---------------------------------|----:|
|From:|The username who sent this message|
|Sender:|The group that username belongs to|
|Newsgroups:|The broad subject field of the email.|
|Subject:|The subject of the message.|
|Write the SFT response here. First, Prefix the first sentence with > to signify that it is a Reasoning sentence.||
|--|The stop tokens|

```
From:user
Sender:usergroup
Newsgroups:motorskills.papercraft
Subject:Paper airplanes

>Provide detailed steps on building a paper airplane.

Instructions: ...

--
```

For finetuning, your data should be in the .mbox format.