---
language:
  - en
license: mit
tags:
  - text-generation
  - causal-lm
  - pretrained
datasets:
  - Solenopsisbot/real-slop
metrics:
  - perplexity
library_name: transformers
pipeline_tag: text-generation
---

# Slopiest-49M

A 49M parameter causal language model trained from scratch on the [real-slop](https://huggingface.co/datasets/Solenopsisbot/real-slop) dataset. Using 2xT4s for free on Kaggle.

## Details

| Property | Value |
|---|---|
| Parameters | ~49M |
| Layers | 6 |
| Attention heads | 6 |
| Embedding size | 384 |
| Context length | 256 tokens |
| Vocabulary | GPT-2 BPE (50257 tokens) |
| Tokenizer | GPT-2 |

## How to Use
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("rudyon/Slopiest-49M")
model = AutoModelForCausalLM.from_pretrained("rudyon/Slopiest-49M", trust_remote_code=True)

inputs = tokenizer("Hello!", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.8, do_sample=True)
print(tokenizer.decode(outputs[0]))
```

## NSFW

As a warning: The dataset used to train this model contains NSFW chats. None of that was filtered out.