README.md

---
pipeline_tag: text-generation
license: mit
base_model: openai-community/openai-gpt
library_name: kerasformers
language:
- en
tags:
- keras
- kerasformers
- gpt
- openai-gpt
- text-generation
- pytorch
- jax
- tf
---

# Run GPT with Keras 3: JAX, PyTorch, or TensorFlow

[![GitHub](https://img.shields.io/badge/GitHub-KerasFormers-black?logo=github)](https://github.com/IMvision12/KerasFormers) [![Docs](https://img.shields.io/badge/Docs-GPT-blue)](https://imvision12.github.io/KerasFormers/gpt/)

# kerasformers/gpt

Paper: [Improving Language Understanding by Generative Pre-Training (Radford et al., 2018)](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)

GPT (the original GPT-1) is OpenAI's first generative pre-trained transformer: a
12-layer decoder-only model with learned position embeddings (512-token context),
`gelu_new` activations, and a byte-pair-encoding tokenizer, trained on BookCorpus.
This is the **117M** base completion model (no chat template).

For more details, see the upstream [model card](https://huggingface.co/openai-community/openai-gpt).

Pure-**Keras 3** conversion of [`openai-community/openai-gpt`](https://huggingface.co/openai-community/openai-gpt) for
[kerasformers](https://github.com/IMvision12/KerasFormers). One implementation runs unmodified on
**TensorFlow / Torch / JAX**.

## Quick start

```python
import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from kerasformers.models.gpt import GptTextGenerate, GptTokenizer

model = GptTextGenerate.from_weights("kerasformers/gpt")
tokenizer = GptTokenizer.from_weights("kerasformers/gpt")

inputs = tokenizer("the meaning of life is")
outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0]))
```

## Tips

- Set `KERAS_BACKEND` **before** importing Keras / kerasformers.
- Context length is 512 tokens; this is a base completion model, not
  instruction-tuned.
- See the [GPT docs](https://imvision12.github.io/KerasFormers/gpt/) and [Loading Weights](https://imvision12.github.io/KerasFormers/loading_weights/).
- Upstream safetensors still work via the `hf:` prefix, e.g.
  `GptTextGenerate.from_weights("hf:openai-community/openai-gpt")`.

## Special Thanks

A huge thank you to the OpenAI GPT authors for creating and releasing this model.

License: MIT, inherited from the upstream OpenAI GPT release.