metadata
language:
- en
license: mit
tags:
- text-generation
- causal-lm
- pretrained
datasets:
- Solenopsisbot/real-slop
metrics:
- perplexity
library_name: transformers
pipeline_tag: text-generation
Slopiest-49M
A 49M parameter causal language model trained from scratch on the real-slop dataset. Using 2xT4s for free on Kaggle.
Details
| Property | Value |
|---|---|
| Parameters | ~49M |
| Layers | 6 |
| Attention heads | 6 |
| Embedding size | 384 |
| Context length | 256 tokens |
| Vocabulary | GPT-2 BPE (50257 tokens) |
| Tokenizer | GPT-2 |
How to Use
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("rudyon/Slopiest-49M")
model = AutoModelForCausalLM.from_pretrained("rudyon/Slopiest-49M", trust_remote_code=True)
inputs = tokenizer("Hello!", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.8, do_sample=True)
print(tokenizer.decode(outputs[0]))
NSFW
As a warning: The dataset used to train this model contains NSFW chats. None of that was filtered out.