---
license: apache-2.0
tags:
  - pytorch
  - gpt
  - tiny-gpt
  - causal-lm
---
# tiny-gpt-2-1m

This repository contains a pretrained TinyGPT checkpoint published for public use.
This checkpoint is provided for educational and experimentation purposes.

## Artifacts

- `tiny_gpt_checkpoint.pt`: training checkpoint with model and optimizer state
- `tokenizer.model`: SentencePiece tokenizer used for training and generation
- `config.json`: model configuration serialized from the checkpoint
- `training_config.yaml`: training and MLflow settings used for the run

## How to use

Use with Transformers.

Starting with `transformers >= 4.43.0`, you can run conversational inference using the `pipeline` abstraction or by leveraging the `Auto` classes with `generate()`.

Make sure to update your Transformers installation via `pip install --upgrade transformers`.

```python
import torch
import transformers

model_id = "vjkhambe/tiny-gpt-2-1m"
device = 0 if torch.cuda.is_available() else -1
dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32

model = transformers.AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
)
tokenizer = transformers.AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model.generation_config.max_length = None
model.generation_config.max_new_tokens = 64

pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    device=device,
)

print(pipeline("Hey how are you doing today?"))
```

## Training details

- Base package: `tiny_gpt_pretrain`
- Model and training configuration are stored in the checkpoint and `training_config.yaml`
- The exported checkpoint includes optimizer state for continued fine-tuning or evaluation

## License

Released under the Apache-2.0 license.

Target repo: `vjkhambe/tiny-gpt-2-1m`