PaddySeahorse/tinyllama-4bit

TinyLlama-1.1B-Chat-v1.0 – 4-bit NF4

This repository contains a **4-bit quantized version of https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0\*\*.

The model was created as a personal experiment while I was first learning about large language models and model quantization around July–August 2025.

The quantization was performed on Google Colab using an NVIDIA T4 GPU, with Hugging Face Transformers and bitsandbytes.

Model Details

• Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Architecture: TinyLlama / Llama-style causal language model
• Task: Text generation / chat
• Quantization: 4-bit
• Quantization backend: bitsandbytes
• Quantization type: NF4
• Compute dtype: torch.bfloat16
• Original model: https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0

Quantization

The model was loaded using the following BitsAndBytesConfig:

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4"
)

It was then loaded with:

model = AutoModelForCausalLM.from_pretrained(
    "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    quantization_config=quantization_config,
    device_map="auto"
)

and uploaded to the Hugging Face Hub with model.push_to_hub().

Original quantization script

For reproducibility and historical reference, the relevant script used to create this repository was:

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from huggingface_hub import login
from accelerate import Accelerator
import torch

# Authenticate with Hugging Face.
# Do not hard-code access tokens in public scripts.
login(token="YOUR_HF_TOKEN")

accelerator = Accelerator()

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4"
)

model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=quantization_config,
    device_map="auto"
)

model = accelerator.prepare(model)

repo_id = "PaddySeahorse/tinyllama-4bit"

model.push_to_hub(
    repo_id,
    commit_message="Push 4-bit quantized TinyLlama"
)

tokenizer.push_to_hub(
    repo_id,
    commit_message="Push tokenizer"
)

Usage

This repository was produced using Hugging Face Transformers and bitsandbytes. A typical loading configuration is:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

repo_id = "PaddySeahorse/tinyllama-4bit"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4"
)

tokenizer = AutoTokenizer.from_pretrained(repo_id)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    quantization_config=quantization_config,
    device_map="auto",
)

For chat-style inference, refer to the usage instructions and chat template of the original TinyLlama model.

Intended Use

This repository was created primarily as a learning experiment for:

• exploring LLM quantization;
• learning the Hugging Face Transformers ecosystem;
• experimenting with bitsandbytes 4-bit loading;
• reducing the memory required to run TinyLlama;
• testing model deployment and sharing through the Hugging Face Hub.

It is not an independently trained or fine-tuned model. The underlying model and its capabilities originate from TinyLlama/TinyLlama-1.1B-Chat-v1.0.

Limitations

This repository should be considered an experimental quantization artifact.

No systematic evaluation was performed after quantization, so no claims are made regarding accuracy, perplexity, benchmark performance, or output quality relative to the original model.

The repository was produced by loading the base model through the bitsandbytes 4-bit quantization path and subsequently uploading it. It should therefore not be confused with separately converted GPTQ, AWQ, or GGUF quantizations.

Compatibility may depend on the versions of transformers, accelerate, bitsandbytes, PyTorch, and the available hardware.

For the original model's intended uses, limitations, training information, and other details, please refer to the https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0.

Hardware

The original quantization experiment was performed in Google Colab with an NVIDIA T4 GPU.

Exact package versions from the original July–August 2025 environment were not recorded.

Acknowledgements

All credit for the base model goes to the TinyLlama project and its contributors.

Base model:

• https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0

This repository only provides the experimental 4-bit quantized artifact created from that model.

License

Please refer to the license and usage terms of the original TinyLlama/TinyLlama-1.1B-Chat-v1.0 model. This repository does not introduce a new license for the underlying model.