|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
tags: |
|
|
- text-generation-inference |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
 |
|
|
|
|
|
## GPT-Usenet-2 |
|
|
An 81-million parameter LLM using GPT-2 encodings. |
|
|
Trained using 10GB of USENET posts along with over 1 GB of miscellaneous BBS posts, digitized books, and text documents. |
|
|
Supervised fine-tuning should be performed before use. |
|
|
|
|
|
## Purpose of GPT-Usenet-2 |
|
|
LLMs are all currently focused on becoming larger and larger, able to do more and more. However, this just makes them jack of all trades, master of none. GPT-Usenet takes a different approach. Instead of trying to do everything perfectly, GPT-Usenet offers a digital stem cell, which can then be finetuned into a single, specialized role and run in parallel with copies of itself. |
|
|
|
|
|
## Technical Information |
|
|
| | | |
|
|
|---------------------------------|----:| |
|
|
|Layers |10| |
|
|
|Heads |10| |
|
|
|Embeddings |640| |
|
|
|Context Window |1024 tokens| |
|
|
|Tokenizer |GPT-2 BPE| |
|
|
|
|
|
|
|
|
## Training Information |
|
|
| | | |
|
|
|---------------------------------|----:| |
|
|
|Training Loss |around 2.0254| |
|
|
|Validation Loss |around 1.9795| |
|
|
|Device |Google Colab L4, Google Colab A100| |
|
|
|Training Time |16 Hours| |
|
|
|
|
|
|
|
|
## Example Syntax |
|
|
|
|
|
| | | |
|
|
|---------------------------------|----:| |
|
|
|From:|The username who sent this message| |
|
|
|Sender:|The group that username belongs to| |
|
|
|Newsgroups:|The broad subject field of the email.| |
|
|
|Subject:|The subject of the message.| |
|
|
|Write the SFT response here. First, Prefix the first sentence with > to signify that it is a Reasoning sentence.|| |
|
|
|--|The stop tokens| |
|
|
|
|
|
``` |
|
|
From:user |
|
|
Sender:usergroup |
|
|
Newsgroups:motorskills.papercraft |
|
|
Subject:Paper airplanes |
|
|
|
|
|
>Provide detailed steps on building a paper airplane. |
|
|
|
|
|
Instructions: ... |
|
|
|
|
|
-- |
|
|
``` |
|
|
|
|
|
For finetuning, your data should be in the .mbox format. |
