AfriSpeech/african-corpus-jw
Viewer • Updated • 2.26M • 2.15k
Afro-Awesome-Align is the first neural word and phrase alignment model specifically fine-tuned across 693 African languages and English. Built on awesome-align (Dou & Neubig, EACL 2021), it extracts dense, high-precision token and multi-word phrase alignments between English and African language parallel texts.
pip install awesome-align transformers torch
Prepare an input file corpus.txt with tab-separated or pipe-separated (|||) lines:
the holy spirit came upon them . ||| honhom kronkron no baa wɔn so .
the high priest questioned him . ||| ɔsɔfo panin no bisaa no .
Run alignment:
awesome-align \
--model_name_or_path AfriSpeech/afro-awesome-align \
--data_file corpus.txt \
--output_file alignments.txt \
--output_word_file aligned_words.txt \
--extraction softmax \
--batch_size 32
import torch
from transformers import AutoModel, AutoTokenizer
model_name = "AfriSpeech/afro-awesome-align"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name).cuda().eval()
src_sent = "the holy spirit came upon them ."
tgt_sent = "honhom kronkron no baa wɔn so ."
# Tokenize and align
src_tokens = src_sent.split()
tgt_tokens = tgt_sent.split()
sent_pair = f"{src_sent} ||| {tgt_sent}"
# Alignments can be extracted directly using the awesome-align pipeline
If you use this model or dataset in your research or applications, please cite:
@inproceedings{dou2021word,
title={Word Alignment by Fine-tuning Embeddings on Parallel Corpora},
author={Dou, Zi-Yi and Neubig, Graham},
booktitle={EACL},
year={2021}
}