t5-smaller

I made this as a smaller, more optimized version of google/flan-t5-small. The weights use bitsandbytes NF4 4-bit quantization with double quantization and FP16 compute.

The name refers to the checkpoint size. This is still the original FLAN-T5 Small architecture: no layers were removed, and it still has roughly 77 million parameters, but it is around 61.7% smaller.

What changed

	Original FLAN-T5 Small	This checkpoint
Stored weights	307.9 MB	118.1 MB
Architecture	FLAN-T5 Small	FLAN-T5 Small
Weight format	FP32	NF4 4-bit

That makes the weight file about 61.7% smaller. This number only describes file size; runtime memory and speed depend on your GPU, PyTorch build, and bitsandbytes version.

Also, much less RAM is used

I did not keep the exact base-model commit or the original conversion environment. If you need a fully reproducible build, treat this checkpoint as a starting point rather than a reference conversion.

Install

Install a PyTorch build that matches your CUDA setup, then run:

pip install transformers accelerate bitsandbytes sentencepiece safetensors

Use it

The model files currently live in optimized-flan-t5-small, so both loader calls need the subfolder argument:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "ShinpacheShimura/t5-smaller"
subfolder = "optimized-flan-t5-small"

tokenizer = AutoTokenizer.from_pretrained(model_id, subfolder=subfolder)
model = AutoModelForSeq2SeqLM.from_pretrained(
    model_id,
    subfolder=subfolder,
    device_map="auto",
)

prompt = "translate English to German: How old are you?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)

print(tokenizer.decode(output[0], skip_special_tokens=True))

I plan to move the model and tokenizer files to the repository root. After that, the two subfolder arguments can be removed.

Benchmark

benchmark.py compares this checkpoint with the FP32 base model on a small, fixed set of prompts:

python benchmark.py \
  --base-model google/flan-t5-small \
  --quantized-model ShinpacheShimura/t5-smaller \
  --quantized-subfolder optimized-flan-t5-small \
  --output-dir benchmark-results

It records loading time, model memory footprint, peak CUDA allocation, generation latency, throughput, and the outputs from both models. Output agreement is useful for spotting quantization changes, but it is not an accuracy score.

I have not published GPU benchmark numbers yet. When I do, I will include the GPU model, package versions, decoding settings, and raw result files here.

Where it may be useful

FLAN-T5 is a text-to-text model, so this checkpoint can be tried on translation, summarization, question answering, and prompted classification. It is mainly intended for experiments where checkpoint size or GPU memory matters.

All this considered, the model is best used when fine-tuned for a specific task, and thanks to it's size, the post trtaining will not consume a lot of resources

The checkpoint is usually faster than the original on most machines. But, small models can lose some of the expected speed benefit to quantization overhead. Check the outputs on your own task before relying on it.

Credits

This checkpoint comes from google/flan-t5-small and keeps its Apache-2.0 license. The original training work is described in:

@article{chung2022scaling,
  title={Scaling Instruction-Finetuned Language Models},
  author={Chung, Hyung Won and Hou, Le and Longpre, Shayne and others},
  journal={arXiv preprint arXiv:2210.11416},
  year={2022}
}

Quantized checkpoint published by ShinpacheShimura.