| | --- |
| | language: en |
| | base_model: google/flan-t5-base |
| | library_name: peft |
| | tags: |
| | - base_model:adapter:google/flan-t5-base |
| | - lora |
| | - transformers |
| | - summarization |
| | - abstractive-summarization |
| | - generated_from_trainer |
| | model-index: |
| | - name: Abstractive Style Summarizer |
| | results: [] |
| | datasets: |
| | - xsum |
| | - cnn_dailymail |
| | - multi_news |
| | license: mit |
| | --- |
| | |
| | # Abstractive Style Summarizer |
| |
|
| | This model is a fine-tuned version of [google/flan-t5-base](https://huggingface.co/google/flan-t5-base) using PEFT (LoRA). It is designed to generate abstractive summaries in three distinct styles: **Harsh** (concise), **Balanced** (standard), and **Detailed** (comprehensive). |
| |
|
| | ## Model Details |
| |
|
| | ### Model Description |
| |
|
| | - **Model type:** Sequence-to-Sequence Transformer (T5) |
| | - **Language(s):** English |
| | - **License:** MIT |
| | - **Finetuned from model:** google/flan-t5-base |
| | - **Training Method:** PEFT (LoRA) |
| |
|
| | ### Model Sources |
| |
|
| | - **Repository:** [Flatten](https://github.com/LityoPS/Flatten) |
| | - **Base Model:** [google/flan-t5-base](https://huggingface.co/google/flan-t5-base) |
| |
|
| | ## Uses |
| |
|
| | ### Direct Use |
| |
|
| | The model interprets a prefixed prompt to determine the style of the summary. |
| | - **Harsh**: Generates very short, punchy summaries (approx. 35% of input length). |
| | - **Balanced**: Generates standard news summaries (approx. 50% of input length). |
| | - **Detailed**: Generates in-depth summaries (approx. 70% of input length). |
| |
|
| | ### Prompt Format |
| |
|
| | The input text should be prefixed with the desired style: |
| | ``` |
| | Summarize {Style}: {Input Text} |
| | ``` |
| | Example: `Summarize Harsh: The Walt Disney Co. announced...` |
| |
|
| | ## Training Details |
| |
|
| | ### Training Data |
| |
|
| | The model was trained on a combined dataset of 12,000 samples, split into 80% Train, 10% Validation, and 10% Test. |
| |
|
| | | Style | Source Dataset | Size | |
| | | :--- | :--- | :--- | |
| | | **Harsh** | [XSum](https://huggingface.co/datasets/xsum) | 4000 | |
| | | **Balanced** | [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) | 4000 | |
| | | **Detailed** | [Multi-News](https://huggingface.co/datasets/multi_news) | 4000 | |
| |
|
| | ### Training Procedure |
| |
|
| | #### Training Hyperparameters |
| |
|
| | - **Learning Rate:** 5e-4 |
| | - **Batch Size:** 4 per device |
| | - **Gradient Accumulation Steps:** 2 |
| | - **Num Epochs:** 5 |
| | - **Optimizer:** AdamW |
| | - **LR Scheduler:** Linear with warmup (ratio 0.05) |
| | - **Mixed Precision:** BF16 |
| |
|
| | #### LoRA Configuration |
| |
|
| | - **r:** 32 |
| | - **lora_alpha:** 64 |
| | - **lora_dropout:** 0.05 |
| | - **target_modules:** ["q", "k", "v", "o"] |
| | - **bias:** "none" |
| | - **task_type:** "SEQ_2_SEQ_LM" |
| | |
| | ### Evaluation Results |
| | |
| | Evaluated on the held-out test set (1,200 samples) at Step 6000. |
| | |
| | | Metric | Score | |
| | | :--- | :--- | |
| | | **ROUGE-1** | 0.3925 | |
| | | **ROUGE-2** | 0.1608 | |
| | | **ROUGE-L** | 0.2776 | |
| | | **Validation Loss** | 0.7824 | |
| | |
| | ## Environmental Impact |
| | |
| | - **Hardware Type:** CUDA-enabled GPU |
| | - **Compute:** LoRA fine-tuning (Parameters: 7M trainable / 254M total) |
| | |
| | ## Framework Versions |
| | |
| | - Datasets==3.6.0 |
| | - Pytorch>=2.5.1 |
| | - Transformers>=4.36.0 |
| | - PEFT>=0.8.0 |
