README.md

---
language:
- so
license: apache-2.0
tags:
- summarization
- headline-generation
- text2text-generation
- somali
- t5
- afriteva
datasets:
- custom
metrics:
- rouge
- sacrebleu
- meteor
- bertscore
pipeline_tag: summarization
---

# Somali Headline Generation (Fine-tuned AfriTeVa v2)

This model generates short, natural Somali news **headlines** from full **article text**. It is a fine-tuned version of [AfriTeVa v2](https://huggingface.co/castorini/afriteva_v2_base), a T5-style sequence-to-sequence model pretrained on African languages, adapted here specifically for Somali headline generation.

## Model Description

- **Base model:** AfriTeVa v2 (T5-style, SentencePiece tokenizer)
- **Task:** Abstractive headline generation (text2text-generation)
- **Language:** Somali (`so`)
- **Input:** A Somali news article (plain text)
- **Output:** A short, cleaned headline summarizing the article

The model was fine-tuned on a dataset of Somali news articles paired with their original headlines, and evaluated using ROUGE, SacreBLEU, METEOR, and BERTScore.

## Intended Uses & Limitations

**Intended uses:**
- Automatic headline suggestion for Somali news articles
- Research on low-resource language summarization/generation
- Educational and non-commercial NLP experimentation

**Limitations:**
- Trained on a specific news domain/style; may not generalize well to other domains (e.g., opinion pieces, social media text)
- May occasionally produce generic or repetitive headlines for very long or unusual articles
- Not intended for use in high-stakes or safety-critical decision-making
- Output quality depends on input article length and clarity; performs best on articles under ~300 tokens (the model's max source length)

## How to Use

### Installation

```bash
pip install transformers torch evaluate sentencepiece
```

### Quick Start

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, T5Tokenizer
import torch

MODEL_NAME = "your-username/your-model-name"  # replace with the actual repo id

# Load tokenizer (SentencePiece preferred; falls back to AutoTokenizer)
try:
    tokenizer = T5Tokenizer.from_pretrained(MODEL_NAME, use_fast=False)
except Exception:
    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=True)

if tokenizer.pad_token is None and tokenizer.eos_token is not None:
    tokenizer.pad_token = tokenizer.eos_token

# Load model
model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_NAME)
model.config.pad_token_id = tokenizer.pad_token_id
model.config.eos_token_id = tokenizer.eos_token_id
if model.config.decoder_start_token_id is None:
    model.config.decoder_start_token_id = tokenizer.pad_token_id

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device).eval()

# Prepare input
article = "Halkan geli qoraalka maqaalkaaga Soomaaliga ah..."
input_text = "headline: " + article

inputs = tokenizer(
    input_text,
    return_tensors="pt",
    truncation=True,
    max_length=300,
).to(device)

# Generate
with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=30,
        min_new_tokens=4,
        num_beams=4,
        no_repeat_ngram_size=3,
        length_penalty=1.3,
        repetition_penalty=1.15,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

headline = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(headline)
```

### Recommended Post-Processing

Raw model output can sometimes contain trailing boilerplate (e.g., source website names, "akhriso", "daawo", stray URLs). We recommend applying a light cleanup step before displaying the headline to end users:

```python
import re

_DOMAINS = r"(?:com|net|org|so|co|tv|fm|info|news|biz|press)"
_TAIL_WORDS = r"(?:akhriso|daawo|dhegayso|dhegeyso|sawirro?|sawiro?|fiiri|video|maqal|muuqaal)"

def clean_headline(h: str) -> str:
    if not isinstance(h, str):
        return h
    h = h.strip()
    h = re.sub(r"https?://\S+", "", h, flags=re.I)                          # remove URLs
    h = re.sub(rf"\b[\w.-]+\.{_DOMAINS}\S*", "", h, flags=re.I)             # remove bare domains
    h = re.sub(rf"\s*[-–—]\s*\b\w+\.{_DOMAINS}\S*$", "", h, flags=re.I)     # trailing "- site.com"
    h = re.sub(rf"\s*\b(?:{_TAIL_WORDS})\b\.?\s*$", "", h, flags=re.I)      # trailing tail words

    if "." in h:
        h = h.split(".", 1)[0]  # keep first sentence only

    h = re.sub(r"\s+([,.;:!?])", r"\1", h)      # fix spacing before punctuation
    h = re.sub(r"([,.;:!?])([^\s])", r"\1 \2", h)  # ensure spacing after punctuation
    h = re.sub(r"\s+", " ", h).strip()

    # remove a dangling trailing connector/particle
    h = re.sub(r"\s+\b(ayaa|ahaa|ka|ku|buu|ayuu|iyo|waxa)\b$", "", h, flags=re.I).strip()

    if h and h[-1] not in ".!?":
        h += "."
    if h:
        h = h[0].lower() + h[1:]
    return h

headline = clean_headline(headline)
print(headline)
```

### Batch Inference

For generating headlines over a full dataset (e.g., a `test_df` with `article` and `headline` columns):

```python
import pandas as pd

@torch.inference_mode()
def batched_generate(texts, batch_size=16):
    preds = []
    for i in range(0, len(texts), batch_size):
        batch_texts = ["headline: " + t for t in texts[i:i + batch_size]]
        enc = tokenizer(
            batch_texts,
            return_tensors="pt",
            padding=True,
            truncation=True,
            max_length=300,
        ).to(device)

        out = model.generate(
            **enc,
            max_new_tokens=30,
            min_new_tokens=4,
            num_beams=4,
            no_repeat_ngram_size=3,
            length_penalty=1.3,
            repetition_penalty=1.15,
            pad_token_id=tokenizer.pad_token_id,
            eos_token_id=tokenizer.eos_token_id,
        )
        for seq in out:
            text = tokenizer.decode(seq, skip_special_tokens=True)
            preds.append(clean_headline(text))
    return preds

test_df = pd.read_csv("your_test_set.csv")  # columns: ["article", "headline"]
predictions = batched_generate(test_df["article"].astype(str).tolist())

test_df["predicted_headline"] = predictions
test_df.to_csv("predictions.csv", index=False)
```

## Evaluation

The model was evaluated on a held-out test set using the following metrics:

| Metric | Description |
|---|---|
| ROUGE-1 / ROUGE-2 / ROUGE-L / ROUGE-Lsum | N-gram and longest-common-subsequence overlap |
| SacreBLEU | BLEU score with standardized tokenization |
| METEOR | Alignment-based metric accounting for synonyms/stems |
| BERTScore (P/R/F1) | Semantic similarity via `xlm-roberta-large` embeddings |


## Training Details

- **Base checkpoint:** `castorini/afriteva_v2_base`
- **Max source length:** 300 tokens
- **Max target length:** 30 tokens
- **Task prefix:** `"headline: "`
- **Generation strategy:** beam search (`num_beams=4`), with repetition penalty and n-gram blocking to reduce repeated phrases


## Citation

If you use this model, please cite the base AfriTeVa v2 model and, optionally, this fine-tuned checkpoint:

```bibtex
@misc{afriteva_v2,
  title  = {AfriTeVa V2: African Text-to-Text Transformer},
  author = {Oladipo, Akintunde and others},
  year   = {2023},
  url    = {https://huggingface.co/castorini/afriteva_v2_base}
}
```

## Acknowledgements

Built on top of [AfriTeVa v2](https://huggingface.co/castorini/afriteva_v2_base) by Castorini, and the Hugging Face `transformers` and `evaluate` libraries.