README.md

---
license: apache-2.0
language:
- en
- de
- es
base_model:
- google/flan-t5-small
pipeline_tag: text-generation
tags:
- text-generation-inference
- summarization
- bitsandbytes
- text2text
- small
- inference
---



# t5-smaller

I made this as a smaller, more optimized version of
[`google/flan-t5-small`](https://huggingface.co/google/flan-t5-small). The
weights use bitsandbytes NF4 4-bit quantization with double quantization and
FP16 compute.

The name refers to the checkpoint size. This is still the original FLAN-T5
Small architecture: no layers were removed, and it still has roughly 77 million
parameters, but it is around 61.7% smaller.

## What changed

| | Original FLAN-T5 Small | This checkpoint |
|---|---:|---:|
| Stored weights | 307.9 MB | 118.1 MB |
| Architecture | FLAN-T5 Small | FLAN-T5 Small |
| Weight format | FP32 | NF4 4-bit |

That makes the weight file about 61.7% smaller. This number only describes file
size; runtime memory and speed depend on your GPU, PyTorch build, and
bitsandbytes version.

Also, much less RAM is used

I did not keep the exact base-model commit or the original conversion
environment. If you need a fully reproducible build, treat this checkpoint as a
starting point rather than a reference conversion.

## Install

Install a PyTorch build that matches your CUDA setup, then run:

```bash
pip install transformers accelerate bitsandbytes sentencepiece safetensors
```

## Use it

The model files currently live in `optimized-flan-t5-small`, so both loader
calls need the `subfolder` argument:

```python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "ShinpacheShimura/t5-smaller"
subfolder = "optimized-flan-t5-small"

tokenizer = AutoTokenizer.from_pretrained(model_id, subfolder=subfolder)
model = AutoModelForSeq2SeqLM.from_pretrained(
    model_id,
    subfolder=subfolder,
    device_map="auto",
)

prompt = "translate English to German: How old are you?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

I plan to move the model and tokenizer files to the repository root. After that,
the two `subfolder` arguments can be removed.

## Benchmark

`benchmark.py` compares this checkpoint with the FP32 base model on a small,
fixed set of prompts:

```bash
python benchmark.py \
  --base-model google/flan-t5-small \
  --quantized-model ShinpacheShimura/t5-smaller \
  --quantized-subfolder optimized-flan-t5-small \
  --output-dir benchmark-results
```

It records loading time, model memory footprint, peak CUDA allocation,
generation latency, throughput, and the outputs from both models. Output
agreement is useful for spotting quantization changes, but it is not an accuracy
score.

I have not published GPU benchmark numbers yet. When I do, I will include the
GPU model, package versions, decoding settings, and raw result files here.

## Where it may be useful

FLAN-T5 is a text-to-text model, so this checkpoint can be tried on translation,
summarization, question answering, and prompted classification. It is mainly
intended for experiments where checkpoint size or GPU memory matters.

All this considered, the model is best used when fine-tuned for a specific task, 
and thanks to it's size, the post trtaining will not consume a lot of resources 

The checkpoint is usually faster than the original on most machines.
But, small models can lose some of the expected speed benefit to quantization
overhead. Check the outputs on your own task before relying on it.

## Credits

This checkpoint comes from
[`google/flan-t5-small`](https://huggingface.co/google/flan-t5-small) and keeps
its Apache-2.0 license. The original training work is described in:

```bibtex
@article{chung2022scaling,
  title={Scaling Instruction-Finetuned Language Models},
  author={Chung, Hyung Won and Hou, Le and Longpre, Shayne and others},
  journal={arXiv preprint arXiv:2210.11416},
  year={2022}
}
```

Quantized checkpoint published by
[`ShinpacheShimura`](https://huggingface.co/ShinpacheShimura).