README.md

metadata

language:
  - so
license: apache-2.0
tags:
  - summarization
  - headline-generation
  - text2text-generation
  - somali
  - t5
  - afriteva
datasets:
  - custom
metrics:
  - rouge
  - sacrebleu
  - meteor
  - bertscore
pipeline_tag: summarization

Somali Headline Generation (Fine-tuned AfriTeVa v2)

This model generates short, natural Somali news headlines from full article text. It is a fine-tuned version of AfriTeVa v2, a T5-style sequence-to-sequence model pretrained on African languages, adapted here specifically for Somali headline generation.

Model Description

• Base model: AfriTeVa v2 (T5-style, SentencePiece tokenizer)
• Task: Abstractive headline generation (text2text-generation)
• Language: Somali (so)
• Input: A Somali news article (plain text)
• Output: A short, cleaned headline summarizing the article

The model was fine-tuned on a dataset of Somali news articles paired with their original headlines, and evaluated using ROUGE, SacreBLEU, METEOR, and BERTScore.

Intended Uses & Limitations

Intended uses:

• Automatic headline suggestion for Somali news articles
• Research on low-resource language summarization/generation
• Educational and non-commercial NLP experimentation

Limitations:

• Trained on a specific news domain/style; may not generalize well to other domains (e.g., opinion pieces, social media text)
• May occasionally produce generic or repetitive headlines for very long or unusual articles
• Not intended for use in high-stakes or safety-critical decision-making
• Output quality depends on input article length and clarity; performs best on articles under ~300 tokens (the model's max source length)

How to Use

Installation

pip install transformers torch evaluate sentencepiece

Quick Start

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, T5Tokenizer
import torch

MODEL_NAME = "your-username/your-model-name"  # replace with the actual repo id

# Load tokenizer (SentencePiece preferred; falls back to AutoTokenizer)
try:
    tokenizer = T5Tokenizer.from_pretrained(MODEL_NAME, use_fast=False)
except Exception:
    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=True)

if tokenizer.pad_token is None and tokenizer.eos_token is not None:
    tokenizer.pad_token = tokenizer.eos_token

# Load model
model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_NAME)
model.config.pad_token_id = tokenizer.pad_token_id
model.config.eos_token_id = tokenizer.eos_token_id
if model.config.decoder_start_token_id is None:
    model.config.decoder_start_token_id = tokenizer.pad_token_id

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device).eval()

# Prepare input
article = "Halkan geli qoraalka maqaalkaaga Soomaaliga ah..."
input_text = "headline: " + article

inputs = tokenizer(
    input_text,
    return_tensors="pt",
    truncation=True,
    max_length=300,
).to(device)

# Generate
with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=30,
        min_new_tokens=4,
        num_beams=4,
        no_repeat_ngram_size=3,
        length_penalty=1.3,
        repetition_penalty=1.15,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

headline = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(headline)

Recommended Post-Processing

Raw model output can sometimes contain trailing boilerplate (e.g., source website names, "akhriso", "daawo", stray URLs). We recommend applying a light cleanup step before displaying the headline to end users:

import re

_DOMAINS = r"(?:com|net|org|so|co|tv|fm|info|news|biz|press)"
_TAIL_WORDS = r"(?:akhriso|daawo|dhegayso|dhegeyso|sawirro?|sawiro?|fiiri|video|maqal|muuqaal)"

def clean_headline(h: str) -> str:
    if not isinstance(h, str):
        return h
    h = h.strip()
    h = re.sub(r"https?://\S+", "", h, flags=re.I)                          # remove URLs
    h = re.sub(rf"\b[\w.-]+\.{_DOMAINS}\S*", "", h, flags=re.I)             # remove bare domains
    h = re.sub(rf"\s*[-–—]\s*\b\w+\.{_DOMAINS}\S*$", "", h, flags=re.I)     # trailing "- site.com"
    h = re.sub(rf"\s*\b(?:{_TAIL_WORDS})\b\.?\s*$", "", h, flags=re.I)      # trailing tail words

    if "." in h:
        h = h.split(".", 1)[0]  # keep first sentence only

    h = re.sub(r"\s+([,.;:!?])", r"\1", h)      # fix spacing before punctuation
    h = re.sub(r"([,.;:!?])([^\s])", r"\1 \2", h)  # ensure spacing after punctuation
    h = re.sub(r"\s+", " ", h).strip()

    # remove a dangling trailing connector/particle
    h = re.sub(r"\s+\b(ayaa|ahaa|ka|ku|buu|ayuu|iyo|waxa)\b$", "", h, flags=re.I).strip()

    if h and h[-1] not in ".!?":
        h += "."
    if h:
        h = h[0].lower() + h[1:]
    return h

headline = clean_headline(headline)
print(headline)

Batch Inference

For generating headlines over a full dataset (e.g., a test_df with article and headline columns):

import pandas as pd

@torch.inference_mode()
def batched_generate(texts, batch_size=16):
    preds = []
    for i in range(0, len(texts), batch_size):
        batch_texts = ["headline: " + t for t in texts[i:i + batch_size]]
        enc = tokenizer(
            batch_texts,
            return_tensors="pt",
            padding=True,
            truncation=True,
            max_length=300,
        ).to(device)

        out = model.generate(
            **enc,
            max_new_tokens=30,
            min_new_tokens=4,
            num_beams=4,
            no_repeat_ngram_size=3,
            length_penalty=1.3,
            repetition_penalty=1.15,
            pad_token_id=tokenizer.pad_token_id,
            eos_token_id=tokenizer.eos_token_id,
        )
        for seq in out:
            text = tokenizer.decode(seq, skip_special_tokens=True)
            preds.append(clean_headline(text))
    return preds

test_df = pd.read_csv("your_test_set.csv")  # columns: ["article", "headline"]
predictions = batched_generate(test_df["article"].astype(str).tolist())

test_df["predicted_headline"] = predictions
test_df.to_csv("predictions.csv", index=False)

Evaluation

The model was evaluated on a held-out test set using the following metrics:

Metric	Description
ROUGE-1 / ROUGE-2 / ROUGE-L / ROUGE-Lsum	N-gram and longest-common-subsequence overlap
SacreBLEU	BLEU score with standardized tokenization
METEOR	Alignment-based metric accounting for synonyms/stems
BERTScore (P/R/F1)	Semantic similarity via xlm-roberta-large embeddings

Training Details

• Base checkpoint: castorini/afriteva_v2_base
• Max source length: 300 tokens
• Max target length: 30 tokens
• Task prefix: "headline: "
• Generation strategy: beam search (num_beams=4), with repetition penalty and n-gram blocking to reduce repeated phrases

Citation

If you use this model, please cite the base AfriTeVa v2 model and, optionally, this fine-tuned checkpoint:

@misc{afriteva_v2,
  title  = {AfriTeVa V2: African Text-to-Text Transformer},
  author = {Oladipo, Akintunde and others},
  year   = {2023},
  url    = {https://huggingface.co/castorini/afriteva_v2_base}
}

Acknowledgements

Built on top of AfriTeVa v2 by Castorini, and the Hugging Face transformers and evaluate libraries.