|
|
--- |
|
|
tags: |
|
|
- text-generation |
|
|
- gpt2 |
|
|
- language-modeling |
|
|
- academic |
|
|
library_name: transformers |
|
|
license: mit |
|
|
datasets: |
|
|
- arxiv |
|
|
metrics: |
|
|
- perplexity |
|
|
--- |
|
|
|
|
|
# AcademicAbstractGenerator: DistilGPT2 Fine-tuned for Scientific Text |
|
|
|
|
|
## 📑 Overview |
|
|
|
|
|
This model is a fine-tuned version of **DistilGPT2**, optimized for the task of generating short, high-quality, and structurally consistent academic abstract drafts. It has been trained exclusively on a corpus of abstracts from arXiv, focusing on fields like Computer Science and Physics. |
|
|
|
|
|
## 🤖 Model Architecture |
|
|
|
|
|
The model utilizes the **GPT-2** decoder-only transformer architecture, offering efficiency and speed due to the Distil model's reduced size. |
|
|
|
|
|
* **Base Model:** `distilgpt2` (a distilled, smaller version of GPT-2). |
|
|
* **Architecture:** Decoder-only transformer stack. |
|
|
* **Layers:** 6 transformer layers. |
|
|
* **Task:** Causal Language Modeling (Text Generation). |
|
|
* **Training Objective:** Minimizing the perplexity on academic text, enabling it to better capture formal structure, complex vocabulary, and typical flow of scientific summaries (Introduction -> Method -> Result -> Conclusion). |
|
|
|
|
|
## 🎯 Intended Use |
|
|
|
|
|
This model is intended for: |
|
|
1. **Drafting:** Assisting researchers in generating initial abstract drafts for new papers. |
|
|
2. **Ideation:** Exploring potential research directions by prompting the model with a topic sentence. |
|
|
3. **Educational Purposes:** Learning about generative model capabilities in a specialized domain. |
|
|
|
|
|
## ⚠️ Limitations |
|
|
|
|
|
* **Factuality:** The model is a text generator, not a knowledge base. Generated content may contain plausible-sounding but **factually incorrect** claims or results. **Human review is mandatory.** |
|
|
* **Length:** Due to its base architecture and training data, it performs best on short sequences (under 256 tokens). |
|
|
* **Overfitting:** May occasionally repeat boilerplate phrases common in academic writing. |
|
|
|
|
|
## 💻 Example Code |
|
|
|
|
|
Use the `TextGenerationPipeline` for drafting abstracts: |
|
|
|
|
|
```python |
|
|
from transformers import pipeline, set_seed |
|
|
|
|
|
set_seed(42) |
|
|
|
|
|
# Load the model and tokenizer |
|
|
generator = pipeline('text-generation', model='[YOUR_HF_USERNAME]/AcademicAbstractGenerator') |
|
|
|
|
|
prompt = "We propose a novel attention mechanism for transformer models that significantly improves training efficiency." |
|
|
|
|
|
# Generate a 150-token abstract draft |
|
|
output = generator( |
|
|
prompt, |
|
|
max_length=150, |
|
|
num_return_sequences=1, |
|
|
temperature=0.7, |
|
|
do_sample=True, |
|
|
truncation=True |
|
|
) |
|
|
|
|
|
print(output[0]['generated_text']) |
