--- language: - la license: mit tags: - latin - token-classification - inverse-text-normalization - casing - punctuation - capitalization - punctuation-restoration - asr-post-processing - classical-latin pipeline_tag: token-classification datasets: - njand/latin-asr-post-processing-dataset model-index: - name: Latin ASR Post-Processor results: - task: type: token-classification name: Inverse Text Normalization dataset: type: njand/latin-asr-post-processing-dataset name: Latin ASR Post-Processing Dataset metrics: - name: Macro F1 type: f1 value: 0.6523 - name: Overall Accuracy type: accuracy value: 0.9216 - name: Precision type: precision value: 0.6228 - name: Recall type: recall value: 0.6943 base_model: - latincy/latin-bert --- # Latin ASR Post-Processor (Casing & Punctuation Restoration) An Inverse Text Normalization (ITN) transformer model fine-tuned to convert unformatted, raw Latin Automatic Speech Recognition (ASR) outputs into fully formatted, classical Latin text. It simultaneously restores capitalization and trailing punctuation using a **14-class composite sequence labeling schema**. --- ### 📌 Quick Links - **Live Demo:** [Gradio Interface](https://huggingface.co/spaces/njand/latin-asr-demo) - **Source Code:** [GitHub Repository](https://github.com/njand/latin-asr-postprocess) - **Base Model:** [`latincy/latin-bert`](https://huggingface.co/latincy/latin-bert) - **Dataset:** [`njand/latin-asr-post-processing-dataset`](https://huggingface.co/datasets/njand/latin-asr-post-processing-dataset) --- ## 🛠️ Pipeline Architecture & Preprocessing This model is intended to be used directly downstream of the acoustic model [`njand/wav2vec2-xls-r-latin`](https://huggingface.co/njand/wav2vec2-xls-r-latin). Because raw ASR models emit stream-of-consciousness text (lowercased, space-separated, and unpunctuated), the text must pass through an input normalization pipeline before being fed into this model for casing and punctuation restoration. ```text +-----------------------+ +-------------------------------+ +-------------------------------+ | Raw Audio Waveform | --> | njand/wav2vec2-xls-r-latin | --> | Preprocessing & Normalization | +-----------------------+ +-------------------------------+ +-------------------------------+ | v +-----------------------+ +-------------------------------+ +-------------------------------+ | Formatted Text Output | <-- | Latin ASR Post-Processor | <-- | Custom CLTK Tokenization | +-----------------------+ +-------------------------------+ +-------------------------------+ ``` ### Input Preprocessing Requirements To prepare raw transcript outputs for inference, apply the following sequence of transformations: 1. **Macron Stripping:** Remove all vowel length diacritics (e.g., *ā, ē, ī, ō, ū, ȳ* → *a, e, i, o, u, y*). 2. **Orthographic Standardization:** Standardize consonant/vowel variants: * Convert *j* → *i* and *v* → *u*. * Handle orthographic exceptions (e.g., *ejicio* → *eicio*). 3. **Custom CLTK Word Tokenization:** Run the normalized string through a version of the **[CLTK (Classical Language Toolkit v0)](https://github.com/cltk/cltk/tree/v0/cltk/tokenize/latin)** Latin word tokenizer. > *Note:* Because official CLTK v0 tokenization scripts are unmaintained, a bespoke implementation of the tokenizer was executed dynamically during training preprocessing rather than being pre-applied to the static dataset. --- ## 🚀 Quickstart & Inference Utility Below is a complete Python script demonstrating how to prepare raw ASR output and run inference using the post-processing pipeline. ```python from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline PUNCT_MAP = { "NONE": "", "COMMA": ",", "PERIOD": ".", "SEMICOLON": ";", "COLON": ":", "QUESTION": "?", "EXCLAMATION": "!", } def format_token(word: str, tag: str) -> str: """Applies composite ITN tag (e.g., 'TITLE_COMMA') to a word token.""" parts = tag.split("_") if len(parts) != 2: return word casing, punct = parts[0], parts[1] if casing == "TITLE": word = word.capitalize() elif casing == "LOWER": word = word.lower() return f"{word}{PUNCT_MAP.get(punct, '')}" def restore_latin_text(pipe, raw_text: str) -> str: """Runs inference and reconstructs formatted Latin text.""" predictions = pipe(raw_text, aggregation_strategy="first") formatted_words = [ format_token(pred["word"].strip(" "), pred["entity_group"]) for pred in predictions ] return " ".join(formatted_words) # 1. Load pipeline model_id = "njand/latin-asr-postprocessor" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForTokenClassification.from_pretrained(model_id) itn_pipe = pipeline("token-classification", model=model, tokenizer=tokenizer) # 2. Test reconstruction with preprocessed ASR output raw_asr_input = "gallia est omnis divisa in partes tres quarum unam incolunt belgae" print(restore_latin_text(itn_pipe, raw_asr_input)) # Output: "Gallia est omnis divisa in partes tres, quarum unam incolunt Belgae." ``` --- ## 🏷️ Composite Label Schema Target labels utilize a **14-class composite sequence schema** that pairs Casing state with Trailing Punctuation state: $$\text{Label} = \text{Casing} \times \text{Punctuation}$$ * **Casing Tags (2):** `LOWER`, `TITLE` * **Punctuation Tags (7):** `NONE`, `COMMA`, `PERIOD`, `SEMICOLON`, `COLON`, `QUESTION`, `EXCLAMATION` --- ## ⚙️ Model Variants & Optimization To facilitate production deployment on CPU-based infrastructure, this repository provides the model in three formats: | Format | Precision | File Size | Latency (P50) | Recommended Use Case | | :--- | :--- | :--- | :--- | :--- | | **PyTorch** | FP32 | 443 MB | - | Training, fine-tuning, and PyTorch pipelines | | **ONNX** | FP32 | 443 MB | 21.5 ms | **Production (Maximum Accuracy)** | | **ONNX Quantized** | INT8 | **188 MB** | **9.7 ms** | **Low-Latency & Edge CPU** | > **Performance vs. Precision Trade-off:** > While INT8 dynamic quantization yields a **2.2× speedup (P50)** and cuts RAM usage by **57.5%**, top-line accuracy (92.16% → 91.03%) hides a severe drop in macro performance: > * **Macro F1 Collapse:** Drops from **65.19% to 50.00%**. Dynamic weight quantization compresses logit decision boundaries for rare token tags. > * **Punctuation Degradation:** Punctuation F1 falls **10.66 percentage points** (69.09% → 58.43%), causing increased missing or misclassified commas, colons, and sentence boundaries. > * **Casing Stability:** Capitalization F1 remains mostly intact (91.71% → 89.38%). > > **Recommendation:** Use **ONNX FP32** for production pipelines where text formatting and punctuation precision are critical. Use **ONNX INT8** in latency-critical environments where speed and memory constraints outweigh exact punctuation recovery. > --- ## 📊 Benchmarks & Performance (Epoch 7 - Best Checkpoint) Evaluated on a 95/5 train/holdout split across diverse Classical Latin literary and historical corpora. ### Overall Summary Metrics | Metric | Score | | --- | --- | | **Overall Accuracy** | **92.16%** | | **Macro F1** | **0.6523** | | **Precision** | **62.28%** | | **Recall** | **69.43%** | | **Validation Loss** | **0.2592** | ### 🎯 Sub-Task Breakdown | Task | Accuracy | F1 Score | | --- | --- | --- | | **Casing Restoration** | **98.35%** | **0.9171** | | **Punctuation Insertion** | **93.67%** | **0.6908** | --- ## 📈 Training Progression The model was trained over 9 epochs fine-tuning [`latincy/latin-bert`](https://www.google.com/url?sa=E&source=gmail&q=https://huggingface.co/latincy/latin-bert). Model weights from **Epoch 7** were selected based on optimal overall F1. | Epoch | Train Loss | Val Loss | Overall F1 | Overall Acc | Casing Acc | Punct Acc | | --- | --- | --- | --- | --- | --- | --- | | 1 | 0.6284 | 0.2883 | 0.6097 | 91.05% | 98.08% | 92.79% | | 2 | 0.5617 | 0.2701 | 0.6275 | 91.60% | 98.19% | 93.26% | | 3 | 0.5150 | 0.2618 | 0.6388 | 91.91% | 98.27% | 93.50% | | 4 | 0.4855 | 0.2607 | 0.6455 | 92.04% | 98.30% | 93.60% | | 5 | 0.4652 | 0.2579 | 0.6440 | 92.12% | 98.33% | 93.65% | | 6 | 0.4502 | 0.2582 | 0.6462 | 92.15% | 98.34% | 93.66% | | **7** | **0.4351** | **0.2592** | **0.6523** | **92.16%** | **98.35%** | **93.67%** | | 8 | 0.4262 | 0.2598 | 0.6495 | 92.24% | 98.36% | 93.75% | | 9 | 0.4172 | 0.2601 | 0.6507 | 92.21% | 98.37% | 93.71% | --- ## ⚡ Hardware & Environmental Footprint * **Hardware Infrastructure:** NVIDIA L4 GPU via Modal * **Training Time:** 3.82 hours * **Estimated Carbon Emissions:** 0.1797 kg CO₂eq