Instructions to use schaeff/gpt2-xl_vanilla800 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use schaeff/gpt2-xl_vanilla800 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="schaeff/gpt2-xl_vanilla800")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("schaeff/gpt2-xl_vanilla800")
model = AutoModelForCausalLM.from_pretrained("schaeff/gpt2-xl_vanilla800")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use schaeff/gpt2-xl_vanilla800 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "schaeff/gpt2-xl_vanilla800"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "schaeff/gpt2-xl_vanilla800",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/schaeff/gpt2-xl_vanilla800

SGLang

How to use schaeff/gpt2-xl_vanilla800 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "schaeff/gpt2-xl_vanilla800" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "schaeff/gpt2-xl_vanilla800",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "schaeff/gpt2-xl_vanilla800" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "schaeff/gpt2-xl_vanilla800",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use schaeff/gpt2-xl_vanilla800 with Docker Model Runner:
```
docker model run hf.co/schaeff/gpt2-xl_vanilla800
```

Model Card for `schaeff/gpt2-xl_vanilla800`

Associated publication: Transformers Don’t Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability (arXiv TBD)

Associated GitHub: removing-layer-norm

This model is based on openai-community/gpt2-xl and was finetuned on OpenWebText for 800 iterations with 0.5M tokens per iteration. This model has the same architecture as the corresponding gpt-2 model and is being made available for reproducibility of the results reported in the associated publication.

Usage

You can load the model with transformers:

model = GPT2LMHeadModel.from_pretrained("schaeff/gpt2-xl_vanilla800")

Model Collection

This model is part of a collection of LayerNorm-free models. The table below provides links and details.

Evaluation results of LN-free, vanilla fine-tuned, and original GPT-2 models

Reported values are mean cross-entropy losses for 10.2M tokens for The Pile and The Pile filtered and 4.5M tokens for the OpenWebText (WT) validation set. For each model size and dataset, the lowest loss is highlighted in bold, and the loss difference between the LN-free model and the best-performing model is shown in brackets.

Model	FT steps	OWT (val)	The Pile	The Pile-filtered
OpenAI GPT-2 Small original	0	3.1006	2.8450	2.7899
schaeff GPT-2 Small vanilla	300	3.0126	2.8511	2.8112
schaeff GPT-2 Small LN-free	300	3.0797 [+0.0671]	2.8852 [+0.0402]	2.8757 [+0.0858]

OpenAI GPT-2 Medium original	0	2.8145	2.5163	2.5390
schaeff GPT-2 Medium vanilla	500	2.7390	2.5752	2.5724
schaeff GPT-2 Medium LN-free	500	2.7642 [+0.0252]	2.6579 [+0.1416]	2.6352 [+0.0962]

OpenAI GPT-2 Large original	0	2.6623	2.5320	2.4347
schaeff GPT-2 Large vanilla	600	2.6240	2.6233	2.5074
schaeff GPT-2 Large LN-free	600	2.6384 [+0.0144]	2.7504 [+0.2184]	2.5159 [+0.0812]

OpenAI GPT-2 XL original	0	2.5567	2.4436¹	2.3739
schaeff GPT-2 XL vanilla	800	2.4799	2.4673	2.3821
schaeff GPT-2 XL LN-free	800	2.5052 [+0.0253]	130.2197²	2.3992 [+0.0253]

Footnotes:

GPT-2 XL original: Median: 1.0103, 95 Percentile Range: [0.0005, 10.6193], 99.9% Percentile Range [≈0.0000, 43.0064]
GPT-2 XL LN-free: Median: 1.0937, 95 Percentile Range: [0.0004, 10.7548], 99.9% Percentile Range [≈0.0000, 48.6459]

Citation

If you have found our work useful please cite as:

@misc{gpt2layernorm2025,
  author = {Baroni, Luca and Khara, Galvin and Schaeffer, Joachim and Subkhankulov, Marat and Heimersheim, Stefan},
  title = {Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability},
  year = {2025},
  eprint = {2507.02559},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  url = {https://arxiv.org/abs/2507.02559v1}
}

Downloads last month: 3

Safetensors

Model size

2B params

Tensor type

F32

Model tree for schaeff/gpt2-xl_vanilla800

Base model

openai-community/gpt2-xl

Finetuned

(71)

this model

Paper for schaeff/gpt2-xl_vanilla800

Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability

Paper • 2507.02559 • Published Jul 3, 2025

Model Card for schaeff/gpt2-xl_vanilla800