from __future__ import annotations import csv import math import re import unicodedata import zipfile from dataclasses import dataclass from io import BytesIO from pathlib import Path from typing import Iterable, Sequence import numpy as np from PIL import Image @dataclass class ImageConfig: target_height: int | None = None target_width: int | None = None preserve_aspect_ratio: bool = True grayscale: bool = True normalize: bool = True binarize: bool = False @dataclass class TextConfig: unicode_form: str = "NFC" remove_tatweel: bool = True remove_diacritics: bool = False keep_punctuation: bool = True collapse_spaces: bool = True strip_text: bool = True TATWEEL = "\u0640" DIACRITIC_PATTERN = re.compile(r"[\u064b-\u065f\u0670]") SPACE_PATTERN = re.compile(r"\s+") ARABIC_PUNCTUATION = "،؛؟" ASCII_PUNCTUATION = r"!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~" PUNCTUATION_PATTERN = re.compile("[" + re.escape(ASCII_PUNCTUATION + ARABIC_PUNCTUATION) + "]") AHCD_CLASS_LABELS = { 1: "alef", 2: "beh", 3: "teh", 4: "theh", 5: "jeem", 6: "hah", 7: "khah", 8: "dal", 9: "thal", 10: "reh", 11: "zain", 12: "seen", 13: "sheen", 14: "sad", 15: "dad", 16: "tah", 17: "zah", 18: "ain", 19: "ghain", 20: "feh", 21: "qaf", 22: "kaf", 23: "lam", 24: "meem", 25: "noon", 26: "heh", 27: "waw", 28: "yeh", } AHCD_FILE_PATTERN = re.compile(r"(?:train|test)/id_(?P\d+)_label_(?P