| # t5-small Quantized Model for Text Summarization on Reddit-TIFU dataset | |
| This repository hosts a quantized version of the t5-small model, fine-tuned for text summarization using the Reddit-TIFU dataset. The model has been optimized using FP16 quantization for efficient deployment without significant accuracy loss. | |
| ## Model Details | |
| - **Model Architecture:** t5-small(short version) | |
| - **Task:** Text generation | |
| - **Dataset:** Reddit-TIFU (Hugging Face Datasets) | |
| - **Quantization:** Float16 | |
| - **Fine-tuning Framework:** Hugging Face Transformers | |
| --- | |
| ## Installation | |
| ```bash | |
| pip install datasets transformers rouge-score evaluate | |
| ``` | |
| --- | |
| ## Loading the Model | |
| ```python | |
| from transformers import AutoModelForSeq2SeqLM, AutoTokenizer | |
| import torch | |
| # Load tokenizer and model | |
| device = 'cuda' if torch.cuda.is_available() else 'cpu' | |
| model_name = "t5-small" | |
| tokenizer = AutoTokenizer.from_pretrained(model_name) | |
| model = AutoModelForSeq2SeqLM.from_pretrained(model_name).to(device) | |
| # Define test sentences | |
| new_text = """ | |
| Today I was late to my morning meeting because I spilled coffee all over my laptop. | |
| Then I realized my backup laptop was also out of battery. | |
| Eventually joined from my phone, only to find out the meeting was cancelled. | |
| """ | |
| # Generate | |
| def generate_summary(text): | |
| inputs = tokenizer( | |
| text, | |
| return_tensors="pt", | |
| max_length=512, | |
| truncation=True | |
| ).to(device) | |
| summary_ids = model.generate( | |
| inputs["input_ids"], | |
| max_length=100, | |
| min_length=5, | |
| num_beams=4, | |
| length_penalty=2.0, | |
| early_stopping=True | |
| ) | |
| summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True) | |
| return summary | |
| ``` | |
| --- | |
| ## Performance Metrics | |
| - **Rouge1:** 19.590 | |
| - **Rouge2:** 4.270 | |
| - **Rougel:** 16.390 | |
| - **Rougelsum:** 16.800 | |
| --- | |
| ## Fine-Tuning Details | |
| ### Dataset | |
| The dataset is sourced from Hugging Face’s `Reddit-TIFU` dataset. It contains 79,000 reddit post and their summaries. | |
| The original training and testing sets were merged, shuffled, and re-split using an 90/10 ratio. | |
| ### Training | |
| - **Epochs:** 3 | |
| - **Batch size:** 8 | |
| - **Learning rate:** 2e-5 | |
| - **Evaluation strategy:** `epoch` | |
| --- | |
| ## Quantization | |
| Post-training quantization was applied using PyTorch’s `half()` precision (FP16) to reduce model size and inference time. | |
| --- | |
| ## Repository Structure | |
| ```python | |
| . | |
| ├── quantized-model/ # Contains the quantized model files | |
| │ ├── config.json | |
| │ ├── model.safetensors | |
| │ ├── tokenizer_config.json | |
| │ ├── spiece.model | |
| │ └── special_tokens_map.json | |
| │ ├── generation_config.jason | |
| │ └── tokenizer.json | |
| ├── README.md # Model documentation | |
| ``` | |
| --- | |
| ## Limitations | |
| - The model is trained specifically for text summarization on reddit posts | |
| - FP16 quantization may result in slight numerical instability in edge cases. | |
| --- | |
| ## Contributing | |
| Feel free to open issues or submit pull requests to improve the model or documentation. | |