Slopiest-49M

A 49M parameter causal language model trained from scratch on the real-slop dataset. Using 2xT4s for free on Kaggle.

Details

Property	Value
Parameters	~49M
Layers	6
Attention heads	6
Embedding size	384
Context length	256 tokens
Vocabulary	GPT-2 BPE (50257 tokens)
Tokenizer	GPT-2

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("rudyon/Slopiest-49M")
model = AutoModelForCausalLM.from_pretrained("rudyon/Slopiest-49M", trust_remote_code=True)

inputs = tokenizer("Hello!", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.8, do_sample=True)
print(tokenizer.decode(outputs[0]))

NSFW

As a warning: The dataset used to train this model contains NSFW chats. None of that was filtered out.