| language: | |
| - en | |
| license: mit | |
| tags: | |
| - text-generation | |
| - causal-lm | |
| - pretrained | |
| datasets: | |
| - Solenopsisbot/real-slop | |
| metrics: | |
| - perplexity | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| # Slopiest-49M | |
| A 49M parameter causal language model trained from scratch on the [real-slop](https://huggingface.co/datasets/Solenopsisbot/real-slop) dataset. Using 2xT4s for free on Kaggle. | |
| ## Details | |
| | Property | Value | | |
| |---|---| | |
| | Parameters | ~49M | | |
| | Layers | 6 | | |
| | Attention heads | 6 | | |
| | Embedding size | 384 | | |
| | Context length | 256 tokens | | |
| | Vocabulary | GPT-2 BPE (50257 tokens) | | |
| | Tokenizer | GPT-2 | | |
| ## How to Use | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("rudyon/Slopiest-49M") | |
| model = AutoModelForCausalLM.from_pretrained("rudyon/Slopiest-49M", trust_remote_code=True) | |
| inputs = tokenizer("Hello!", return_tensors="pt") | |
| outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.8, do_sample=True) | |
| print(tokenizer.decode(outputs[0])) | |
| ``` | |
| ## NSFW | |
| As a warning: The dataset used to train this model contains NSFW chats. None of that was filtered out. | |