README.md

---
license: apache-2.0
datasets:
- abisee/cnn_dailymail
language:
- en
metrics:
- rouge
- bertscore
base_model:
- allenai/led-base-16384
pipeline_tag: summarization
library_name: transformers
tags:
- summarization
- text2text-generation
- text-generation
- encoder-decoder
- led
- longformer
- bart
- abstractive-summarization
- news-summarization
- research-summarization
- document-summarization
- english
- NLP
---
# Research & News AI Summarizer

![Research & News AI Summarizer](media/model.png)

**Model Type:** NLP / Text Summarization


---

## Model Description

Research & News AI Summarizer is a fine-tuned Longformer Encoder-Decoder (LED) model optimized for abstractive summarization of long-form research articles and news content. It is trained using a custom 3-stage curriculum learning strategy across multiple versions of the CNN/DailyMail dataset, and is powered by the STAM (Stable Training with Adaptive Momentum) optimizer.

The model supports input sequences of up to 8,192 tokens (base configuration) or 16,384 tokens (large configuration), making it suitable for summarizing lengthy documents that exceed the context limits of standard transformer models.

## Developer

- **Developer:** [Assem Sabry](https://assem.cloud/)
- **GitHub:** [assemsabry/Research-News-AI-Summarizer](https://github.com/assemsabry/Research-News-AI-Summarizer)
- **HuggingFace Model:** [assemsabry/Research-News-AI-Summarizer](https://huggingface.co/assemsabry/Research-News-AI-Summarizer)
- **PyPI (STAM Optimizer):** [stam-optimizer](https://pypi.org/project/stam-optimizer/)
- **STAM GitHub:** [github.com/assemsabry/stam](https://github.com/assemsabry/stam)

---

## STAM Optimizer

This model is trained exclusively with the STAM optimizer, a next-generation adaptive momentum optimizer designed for stable convergence in large-scale NLP training.

![STAM Optimizer](media/stam.png)

STAM dynamically adjusts first-momentum coefficients based on gradient alignment statistics, reducing training instability and improving generalization across long-context summarization tasks.

**Key STAM Hyperparameters:**

| Parameter | Value | Description |
|-----------|-------|-------------|
| `learning_rate` | 1.0e-4 | Initial learning rate |
| `b1_base` | 0.9 | Base first-momentum coefficient |
| `b2` | 0.999 | Second-moment decay rate |
| `weight_decay` | 0.01 | Decoupled weight decay |
| `adapt_strength` | 0.2 | Momentum adaptation strength [0, 0.5] |

**References:**
- Paper: [Stable Training with Adaptive Momentum](https://tokenai.cloud/research/stam)
- Author: [Assem Sabry](https://assem.cloud/)
- Repository: https://github.com/assemsabry/stam
- Package: https://pypi.org/project/stam-optimizer/
- Research: https://tokenai.cloud/research/stam

---

## Model Specifications

| Attribute | Value |
|-----------|-------|
| Base Model | [allenai/led-base-16384](https://huggingface.co/allenai/led-base-16384) |
| Architecture | Longformer Encoder-Decoder (LED) |
| Context Length | 8,192 tokens (base) / 16,384 tokens (large) |
| Max Summary Length | 512 tokens (base) / 768 tokens (large) |
| Min Summary Length | 64 tokens (base) / 80 tokens (large) |
| Vocabulary Size | 50,265 |
| Parameters | ~162M |
| Optimizer | STAM (Stable Training with Adaptive Momentum) |
| Training Type | Full Fine-Tuning (no LoRA / PEFT) |
| Precision | FP16 mixed precision |
| Gradient Checkpointing | Enabled |

---

## Training Details

### Hardware

- **GPUs:** 2x NVIDIA Tesla T4
- **Total VRAM:** 32 GB (16 GB per GPU)
- **Environment:** CUDA 12.1, PyTorch 2.0+

### Training Regime

- **Effective Batch Size:** 32 (1 per device x 16 gradient accumulation steps x 2 GPUs)
- **Total Training Time:** ~26 hours
- **Seed:** 42
- **Warmup Ratio:** 3%
- **Max Gradient Norm:** 1.0
- **Early Stopping Patience:** 2 evaluations

### Training Stages (Curriculum Learning)

| Stage | Dataset Version | Train Samples | Max Steps | Eval Steps |
|-------|----------------|---------------|-----------|------------|
| Stage 1 | CNN/DailyMail v1.0.0 | 50,000 | 1,562 | 500 |
| Stage 2 | CNN/DailyMail v2.0.0 | 30,000 | 937 | 500 |
| Stage 3 | CNN/DailyMail v3.0.0 | 30,000 | 937 | 500 |
| **Total** | | **110,000** | **3,436** | |

### Training History

The model was trained over approximately 26 hours using the STAM optimizer with adaptive momentum. The training loss curve demonstrated stable convergence throughout all three curriculum stages.

**Loss Progression:**

| Stage | Step Range | Initial Loss | Final Loss | Notes |
|-------|-----------|-------------|-----------|-------|
| Stage 1 | 1 - 1,562 | 2.82 | 1.74 | Warmup completed at step 46. Loss stabilized after step 400. |
| Stage 2 | 1,563 - 2,499 | 1.68 | 1.45 | Curriculum shift to v2.0.0 data. Minor spike at step 1,600 then smooth descent. |
| Stage 3 | 2,500 - 3,436 | 1.42 | 1.38 | Final refinement on v3.0.0. Convergence reached by step 3,200. |

**Validation ROUGE-Lsum Progression:**

| Stage | Initial ROUGE-Lsum | Final ROUGE-Lsum | Best Checkpoint Step |
|-------|-------------------|-------------------|---------------------|
| Stage 1 | 28.45 | 35.12 | Step 1,500 |
| Stage 2 | 35.80 | 38.94 | Step 2,450 |
| Stage 3 | 39.10 | 42.36 | Step 3,350 |

**Training Throughput:**

- Average step time: ~27 seconds
- Peak GPU memory usage: ~14.9 GB per GPU
- Total tokens processed: ~898M (input + target)

### Dataset Details

- **Source:** [abisee/cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail)
- **Versions Used:** 1.0.0, 2.0.0, 3.0.0
- **Splits:** train (for training), validation (for evaluation), test (for final testing)
- **Text Normalization:** HTML stripping, whitespace normalization, sentence-level validation
- **Filtering:** Articles with fewer than 120 words or summaries outside the 8-350 word range are excluded

### Data Preprocessing

```python
# Example preprocessing pipeline
from transformers import LEDTokenizer

tokenizer = LEDTokenizer.from_pretrained("assemsabry/Research-News-AI-Summarizer")

# Article tokenization (max 8,192 tokens)
inputs = tokenizer(article, max_length=8192, truncation=True)

# Summary tokenization (max 512 tokens)
labels = tokenizer(text_target=summary, max_length=512, truncation=True)
```

---

## Evaluation Results

### Test Set Performance (CNN/DailyMail v3.0.0, 500 samples)

| Metric | Score |
|--------|-------|
| ROUGE-1 | 43.82 |
| ROUGE-2 | 20.65 |
| ROUGE-L | 40.28 |
| ROUGE-Lsum | 42.36 |
| BERTScore Precision | 91.24 |
| BERTScore Recall | 90.18 |
| BERTScore F1 | 90.71 |
| Avg Generation Length | 142.3 tokens |

### Performance by Summary Length Bucket

| Length Bucket | ROUGE-1 | ROUGE-2 | ROUGE-L | BERTScore F1 |
|--------------|---------|---------|---------|-------------|
| Short (0-80 words) | 46.12 | 22.85 | 42.65 | 91.85 |
| Medium (80-160 words) | 44.38 | 21.20 | 40.94 | 90.92 |
| Long (160-320 words) | 41.25 | 18.45 | 38.12 | 89.45 |
| Very Long (320+ words) | 38.90 | 16.20 | 35.80 | 88.12 |

---

## Usage

### Quick Start

```python
from transformers import LEDForConditionalGeneration, LEDTokenizer
import torch

model = LEDForConditionalGeneration.from_pretrained(
    "assemsabry/Research-News-AI-Summarizer"
)
tokenizer = LEDTokenizer.from_pretrained(
    "assemsabry/Research-News-AI-Summarizer"
)

article = """
Your long article text here. This model is designed to handle up to 8,192 tokens
of input context, making it suitable for research papers, news articles, and
other long-form content that exceeds the limits of standard BART or T5 models.
"""

inputs = tokenizer(
    article,
    max_length=8192,
    truncation=True,
    return_tensors="pt"
)

# LED global attention mask: attend to the first token globally
global_attention_mask = torch.zeros_like(inputs["input_ids"])
global_attention_mask[:, 0] = 1

summary_ids = model.generate(
    **inputs,
    global_attention_mask=global_attention_mask,
    max_length=512,
    min_length=64,
    num_beams=4,
    length_penalty=2.0,
    no_repeat_ngram_size=3,
    early_stopping=True,
)

summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
print(summary)
```

### Batch Inference

```python
articles = [article1, article2, article3]
inputs = tokenizer(
    articles,
    max_length=8192,
    truncation=True,
    padding=True,
    return_tensors="pt"
)
global_attention_mask = torch.zeros_like(inputs["input_ids"])
global_attention_mask[:, 0] = 1

summary_ids = model.generate(
    **inputs,
    global_attention_mask=global_attention_mask,
    max_length=512,
    num_beams=4,
)
summaries = tokenizer.batch_decode(summary_ids, skip_special_tokens=True)
```

---

## Repository Structure

```
.
├── src/
│   ├── config.py          # Training configuration with large sequence settings
│   ├── dataset.py         # CNN/DailyMail dataset processing
│   ├── metrics.py         # ROUGE and BERTScore evaluation
│   ├── model.py           # LED model loading and setup
│   ├── optimizer.py       # STAM and STAMLite PyTorch optimizers
│   └── trainer.py         # Custom Seq2Seq trainer with STAM
├── scripts/
│   ├── train.py           # Main multi-stage training script
│   └── evaluate.py        # Evaluation and inference script
├── configs/
│   ├── base_config.yaml   # Base training configuration
│   └── large_config.yaml  # Large-scale training configuration
├── tests/
│   └── test_model.py      # Unit tests for components
├── media/
│   └── stam.png           # STAM optimizer diagram
├── requirements.txt       # Python dependencies
├── setup.py              # Package installation
└── README.md             # This file
```

---

## Installation

```bash
git clone https://github.com/assemsabry/Research-News-AI-Summarizer
cd Research-News-AI-Summarizer
pip install -r requirements.txt
```

---

## Training

Set your Hugging Face token as an environment variable:

```bash
export HF_TOKEN="your_token_here"
```

Run multi-stage training:

```bash
python scripts/train.py --output-dir ./outputs --stages 1 2 3
```

Run specific stages only:

```bash
python scripts/train.py --stages 2 3
```

Skip Hub upload (local training only):

```bash
python scripts/train.py --skip-upload
```

### Training with Custom Config

```python
from src.config import Config
from src.model import load_model_and_tokenizer, setup_system
from src.trainer import build_training_args, build_trainer

config = Config()
config.data.max_input_length = 16384
config.data.max_target_length = 768
config.training.gradient_accumulation_steps = 32

setup_system(config)
model, tokenizer = load_model_and_tokenizer(config)
```

---

## Evaluation

Evaluate a trained checkpoint:

```bash
python scripts/evaluate.py \
    --model-path ./outputs/artifacts/stage_3_cnn_v3 \
    --output-dir ./reports \
    --num-samples 50
```

The evaluation script produces:
- `human_evaluation.csv`: Generated summaries with per-sample ROUGE and BERTScore
- `error_analysis.csv`: Length-based error analysis with ratio statistics
- `evaluation_stats.json`: Aggregate metrics by length bucket

---

## Testing

Run unit tests:

```bash
python -m unittest tests/test_model.py
```

Tests cover:
- Text normalization and HTML stripping
- Dataset validation (min/max word counts)
- STAM and STAMLite optimizer initialization and step logic
- Configuration defaults and effective batch size computation

---

## Limitations and Biases

- The model is trained exclusively on English news articles (CNN/DailyMail). Performance on non-English text or highly technical research papers outside the news domain may vary.
- Summaries may inherit biases present in the original CNN/DailyMail dataset.
- The model does not fact-check generated content. Hallucinations can occur on out-of-domain inputs.
- Maximum input length is 8,192 tokens (base) or 16,384 tokens (large). Documents exceeding this length are truncated from the end.

---

## License

Apache 2.0

---

## Citation

If you use this model in your research, please cite:

```bibtex
@misc{research-news-ai-summarizer,
  title={Research & News AI Summarizer: Fine-tuned LED with STAM Optimizer},
  author={Sabry, Assem},
  year={2025},
  howpublished={\url{https://huggingface.co/assemsabry/Research-News-AI-Summarizer}}
}
```

---

## Acknowledgments

- Base model: [allenai/led-base-16384](https://huggingface.co/assemsabry/Research-News-AI-Summarizer/tree/main/datasets) by AllenAI
- Dataset: [CNN/DailyMail](https://huggingface.co/datasets/abisee/cnn_dailymail)
- Optimizer: [STAM](https://github.com/assemsabry/stam) by Assem Sabry