|
|
|
|
|
--- |
|
|
tags: |
|
|
- causal-lm |
|
|
- text-generation |
|
|
- pre-trained |
|
|
- pytorch |
|
|
--- |
|
|
|
|
|
# tinystories-gpt-small |
|
|
|
|
|
This is a custom GPT model **pre-trained from scratch on the TinyStories dataset**. |
|
|
It demonstrates foundational language modeling capabilities and can be used for text generation. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
* **Architecture:** Custom GPT |
|
|
* `n_layer`: 8 |
|
|
* `n_head`: 8 |
|
|
* `n_embd`: 512 |
|
|
* `block_size`: 1024 |
|
|
* `vocab_size`: 50257 |
|
|
* `dropout`: 0.1 |
|
|
* **Pre-training Dataset:** TinyStories (a synthetic dataset of short, simple stories designed to teach language models basic reasoning and coherence). |
|
|
* **Purpose:** This model is a base language model. It has learned to predict the next token in a sequence based on the patterns found in the TinyStories dataset. It is suitable for demonstrating basic generative text capabilities and serves as a foundation for further fine-tuning on specific downstream tasks (e.g., question answering, chatbot). |
|
|
|
|
|
## How to Use (Inference) |
|
|
|
|
|
```python |
|
|
import torch |
|
|
import tiktoken |
|
|
from model import GPT, GPTConfig # Assuming model.py is available or its classes are defined |
|
|
|
|
|
# 1. Define model configuration (must match the trained model's config.json) |
|
|
# You can load this from config.json if you save it, or define it manually |
|
|
config = GPTConfig( |
|
|
vocab_size=50257, |
|
|
block_size=1024, |
|
|
n_layer=8, |
|
|
n_head=8, |
|
|
n_embd=512, |
|
|
dropout=0.1, |
|
|
bias=True |
|
|
) |
|
|
|
|
|
# 2. Initialize the model and load weights |
|
|
model = GPT(config) |
|
|
state_dict = torch.load("pytorch_model.bin", map_location='cpu') # Replace with path to downloaded model |
|
|
model.load_state_dict(state_dict) |
|
|
model.eval() # Set to evaluation mode |
|
|
device = 'cuda' if torch.cuda.is_available() else 'cpu' |
|
|
model.to(device) |
|
|
|
|
|
# 3. Load the tiktoken tokenizer |
|
|
tokenizer = tiktoken.get_encoding("gpt2") |
|
|
EOT_TOKEN_ID = tokenizer.eot_token |
|
|
|
|
|
# 4. Prepare your prompt for text generation |
|
|
prompt_text = "Once upon a time there was a pumpkin." |
|
|
|
|
|
# Encode the prompt |
|
|
allowed_special_tokens = 'all' |
|
|
input_ids = tokenizer.encode(prompt_text, allowed_special=allowed_special_tokens) |
|
|
input_ids_tensor = torch.tensor([input_ids], dtype=torch.long).to(device) |
|
|
|
|
|
# 5. Generate text |
|
|
# Adjust max_new_tokens, temperature, top_k as needed |
|
|
generated_output_ids = model.generate( |
|
|
idx=input_ids_tensor, |
|
|
max_new_tokens=100, # Max length for the generated text |
|
|
temperature=0.7, |
|
|
top_k=50 |
|
|
) |
|
|
|
|
|
# Decode the generated text (excluding the prompt part) |
|
|
generated_text_ids = generated_output_ids[0, len(input_ids):].tolist() |
|
|
generated_text = tokenizer.decode(generated_text_ids) |
|
|
|
|
|
# Clean up any leftover EOT tokens from generation |
|
|
generated_text = generated_text.replace(tokenizer.decode([EOT_TOKEN_ID]), "").strip() |
|
|
|
|
|
print(f"Generated Text: {generated_text}") |
|
|
``` |
|
|
|
|
|
## Limitations and Bias |
|
|
|
|
|
* This model is a relatively small GPT (50.95M parameters) and its generative capabilities are limited by its size and the simplicity of the TinyStories dataset. |
|
|
* It is a base language model and has not been instruction-tuned or fine-tuned for specific tasks like complex question answering or dialogue. Therefore, its responses may be incoherent or non-factual for out-of-distribution prompts. |
|
|
* Like all language models, it may generate biased or incorrect information based on its training data. |
|
|
|
|
|
## License |
|
|
|
|
|
Apache 2.0 |
|
|
|
|
|
|
