| --- |
| language: en |
| license: apache-2.0 |
| base_model: t5-base |
| tags: |
| - summarization |
| - research-paper |
| - seq2seq |
| - t5 |
| - lora |
| - peft |
| datasets: |
| - custom |
| metrics: |
| - rouge |
| - bertscore |
| --- |
| |
| # T5-Base-Summarization |
|
|
| A fine-tuned version of [t5-base](https://huggingface.co/t5-base) for summarizing research papers into concise summaries. This is the first stage of a two-step **Research Paper Simplifier** pipeline. |
|
|
| ## Model Description |
|
|
| This model takes a section of a research paper as input and generates a plain-language summary approximately 1/10th the length of the original text. Fine-tuned using LoRA (PEFT) for parameter-efficient training. |
|
|
| ## Pipeline |
|
|
| ``` |
| Research Paper ──► [T5-Base-Summarization] ──► Summary ──► [T5-Base-Story-Generation] ──► Story |
| ``` |
|
|
| ## Training Details |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | Base model | t5-base | |
| | Task | Summarization | |
| | Max input length | 1024 tokens | |
| | Max target length | 128 tokens | |
| | Learning rate | 3e-5 | |
| | Batch size | 4 | |
| | Gradient accumulation steps | 4 | |
| | Warmup steps | 500 | |
| | Weight decay | 0.01 | |
| | Fine-tuning method | LoRA (r=16, alpha=32, targets: q, v) | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForSeq2SeqLM |
| |
| tokenizer = AutoTokenizer.from_pretrained("harsharajkumar273/T5-Base-Summarization") |
| model = AutoModelForSeq2SeqLM.from_pretrained("harsharajkumar273/T5-Base-Summarization") |
| |
| text = "Your research paper section here..." |
| word_count = len(text.split()) |
| prompt = f"Summarize this part of the research paper to less than {word_count // 10} words:\n{text}" |
| |
| inputs = tokenizer(prompt, return_tensors="pt", max_length=1024, truncation=True) |
| outputs = model.generate(**inputs, max_length=128, num_beams=4) |
| summary = tokenizer.decode(outputs[0], skip_special_tokens=True) |
| print(summary) |
| ``` |
|
|
| ## Evaluation Metrics |
|
|
| Evaluated using ROUGE and BERTScore on a held-out 10% test split. |
|
|
| ## Related Models |
|
|
| - [harsharajkumar273/Bart-Base-Summarization](https://huggingface.co/harsharajkumar273/Bart-Base-Summarization) |
| - [harsharajkumar273/ProphetNet-Large-Summarization](https://huggingface.co/harsharajkumar273/ProphetNet-Large-Summarization) |
| - [harsharajkumar273/T5-Base-Story-Generation](https://huggingface.co/harsharajkumar273/T5-Base-Story-Generation) — next stage |
|
|
