Buckets:
| Name | Size | Uploaded | Xet hash |
|---|---|---|---|
| notebooks | 3 items | ||
| README.md | 6.89 kB xet | 80e9d9ff | |
| esperanto_20_evaluation_probes.json | 3.29 kB xet | eb1ec21a |
Systematic Evaluation of Morphosyntactic & Socio-Linguistic Blind Spots in Frontier Small Language Models: The Esperanto Capability Gap(Fall 2026 Fatima Fellowship)
1. Blind Spot Identification & Motivation (Part 1)
Standard LLM benchmark suites (e.g., MMLU, GSM8K, HumanEval) primarily evaluate English and high-resource natural languages. As a result, small-to-medium open-weight frontier models (0.6B–6B parameter class) exhibit significant, underexplored capability gaps in low-to-medium resource constructed languages like Esperanto.
Despite Esperanto’s completely regular agglutinative morphology and rigid 45-word correlative matrix (Tabelvortoj), state-of-the-art small language models struggle with:
- Directional Accusative Case Shifts (
-nSuffix): Distinguishing static location vs. directional movement across spatial prepositions (e.g., en la domo vs. en la domon). - Recursive Affix Agglutination: Composing and deconstructing words via agglutinative affixes without hallucinating fake roots or gender suffixes.
- Correlative System Matrix Lookup: Maintaining structural integrity across the 5x9 correlative grid.
- Socio-Linguistic Idioms & Movement Culture: Understanding domain-specific jargon cultivated by the Esperanto speech community (e.g., krokodili, Finvenkismo, samideano, Volapūkaĵo).
2. Systematic Evaluation Setup & Multi-Model Comparison (Part 2)
I evaluated 20 systematic linguistic probes across three leading frontier small language models hosted on Hugging Face (within the 0.6B–6B parameter range):
- Model A:
google/gemma-4-E4B-it(4.5B Effective Parameters, 8B total with embeddings) - Model B:
openbmb/MiniCPM5-2B(2.0B Parameters) - Model C:
ibm-granite/granite-3.1-2b-instruct(2.5B Parameters)
All evaluation runs were conducted in Google Colab L4 GPU runtimes using greedy decoding (temperature=0.0, do_sample=False) for exact reproducibility.
Comparative Evaluation Summary Table
| Category | Probe ID & Task | google/gemma-4-E4B-it |
openbmb/MiniCPM5-2B |
ibm-granite/granite-3.1-2b-instruct |
|---|---|---|---|---|
| Directional Accusative | [1/20] "Cat runs into house" (Boundary entry) | La kato iras al la domo (Fails -n case) |
Reasoning loop failure; questions if "run" = paroli | La gato flugis al domo (Spanish leakage: gato, wrong verb) |
| [3/20] "She jumped onto table" | Ŝi saltis sur la tablo (Fails -n case) |
Kiu jumis sur tavla (Hallucinated roots) |
Ela kantante al la tablo (Corrupted syntax) |
|
| [5/20] "Bird flew under bridge" | La birdo flugis sub la ponton (Correct) |
La fonto fliĝis sub la pordo (Fabricated nouns) |
La avena flugas sub la brodo (Fabricated nouns) |
|
| Affix Derivation | [6/20] Deconstruct malsanulejo | Correctly breaks down mal-, san-, -ul-, -ej-, -o | Claims -ulo = "child", -jo = particle | Claims sanulejo derives from Latin ulcerare |
| [8/20] "Female teacher" derivation | Fabricates virinisto (Fails -ino suffix rules) |
Claims feminine noun suffix is -a |
Fabricates docenteja (Spanish leakage) |
|
| [10/20] "Small place where bad people gather" | Multi-word phrase: Malbona-kolekto | Fabricates root malsamul | Fabricates compound: mal-luko-jado-kompa | |
| Correlative Matrix | [11/20] Sequence: kiam, tiam, ĉiam, ___ | Fabricates nuniam |
Fabricates neviĉ |
Claims ne is the correlative |
| [12/20] Correlative for "such / of that kind" | Fabricates taŭa |
Fails matrix lookup; hallucinates kiel | Claims de kio or kio |
|
| [13/20] Fill: Kiu -> Tiu -> Ĉiu -> ___ | Neniu (Correct) |
Questions if words are Chinese/Esperanto | Fabricates niunu |
|
| Community Idioms | [16/20] Define krokodili | "Acting overly dramatic / melodramatic" | "Not a real Esperanto word" | "Someone who is slow to learn" |
| [18/20] Define Volapūkaĵo | "Not a standard Esperanto word" | "Not a standard Esperanto word" | "Language of the world" (Literal translation) | |
| [20/20] Define samideano | "Not a standard Esperanto word" | "Not a standard Esperanto word" | Claims it means "equal-to-all" |
Detailed Failure Mode Taxonomy
- Subword Tokenization & Romance Vocabulary Leakage:
- IBM Granite frequently spills over into Spanish, Italian, and Latin subwords (
gato,corre,docente,skola,brodo), proving that its subword tokenizer merges Esperanto roots into Romance language frequency vectors.
- IBM Granite frequently spills over into Spanish, Italian, and Latin subwords (
- Correlative Table Lookup Collapse:
- All three models fail to navigate the 5x9 correlative matrix. Both Gemma 4 and MiniCPM invent non-existent correlatives (nuniam, taŭa, neviĉ) rather than querying the rigid paradigm (neniam, tia).
- Morphological Agglutination Brittleness:
- While Esperanto allows infinite valid compounding via affixes, models either invent false suffixes (Gemma's virinisto instead of instruistino) or completely fail to recognize foundational affixes like -ino (feminine) or -ej- (place).
- Complete Socio-Linguistic & Cultural Erasure:
- Every tested model fails on community jargon. Terms like krokodili (speaking native language at an Esperanto event), Volapūkaĵo (gibberish/nonsense), and samideano (fellow Esperantist / "same-idea-member") are either declared "non-existent words" or hallucinated with false etymologies.
3. Proposed Technical Path Forward (Part 3)
To close this capability gap in sub-6B models without drastically expanding parameter count, I propose a 3-pillar intervention strategy:
- Targeted Data Curation & Synthetic Data Generation:
- Collect parallel and monolingual corpora from authentic community archives (Tekstaro de Esperanto, Vikipedio, Monato).
- Generate synthetic rule-based contrastive pairs explicitly probing the directional accusative (
-nshift across all prepositions) and systematic correlative substitution matrices.
- Tokenizer Vocabulary Extension:
- Add the 45 correlative words (Tabelvortoj) and standard 40+ Esperanto affixes as atomic tokens into the model's vocabulary, preventing subword fragmentation into Romance language roots.
- PEFT / QLoRA Alignment Tuning:
- Apply QLoRA fine-tuning or Direct Preference Optimization (DPO) using preference pairs that explicitly penalize false derivations (e.g., rejecting virinisto or docenteja in favor of instruistino).
Repository Structure
├── README.md
├── esperanto_20_evaluation_probes.json
└── notebooks/
├── gemma4_e4b_esperanto_eval.ipynb
├── minicpm5_2b_esperanto_eval.ipynb
└── granite_3.1_2b_esperanto_eval.ipynb
- Total size
- 287 kB
- Files
- 5
- Last updated
- Oct 22
- Pre-warmed CDN
- US EU US EU