287 kB
5 files
Updated 28 days ago
Name
Size
notebooks
README.md6.89 kB
xet
esperanto_20_evaluation_probes.json3.29 kB
xet
README.md

Systematic Evaluation of Morphosyntactic & Socio-Linguistic Blind Spots in Frontier Small Language Models: The Esperanto Capability Gap(Fall 2026 Fatima Fellowship)

1. Blind Spot Identification & Motivation (Part 1)

Standard LLM benchmark suites (e.g., MMLU, GSM8K, HumanEval) primarily evaluate English and high-resource natural languages. As a result, small-to-medium open-weight frontier models (0.6B–6B parameter class) exhibit significant, underexplored capability gaps in low-to-medium resource constructed languages like Esperanto.

Despite Esperanto’s completely regular agglutinative morphology and rigid 45-word correlative matrix (Tabelvortoj), state-of-the-art small language models struggle with:

  1. Directional Accusative Case Shifts (-n Suffix): Distinguishing static location vs. directional movement across spatial prepositions (e.g., en la domo vs. en la domon).
  2. Recursive Affix Agglutination: Composing and deconstructing words via agglutinative affixes without hallucinating fake roots or gender suffixes.
  3. Correlative System Matrix Lookup: Maintaining structural integrity across the 5x9 correlative grid.
  4. Socio-Linguistic Idioms & Movement Culture: Understanding domain-specific jargon cultivated by the Esperanto speech community (e.g., krokodili, Finvenkismo, samideano, Volapūkaĵo).

2. Systematic Evaluation Setup & Multi-Model Comparison (Part 2)

I evaluated 20 systematic linguistic probes across three leading frontier small language models hosted on Hugging Face (within the 0.6B–6B parameter range):

  • Model A: google/gemma-4-E4B-it (4.5B Effective Parameters, 8B total with embeddings)
  • Model B: openbmb/MiniCPM5-2B (2.0B Parameters)
  • Model C: ibm-granite/granite-3.1-2b-instruct (2.5B Parameters)

All evaluation runs were conducted in Google Colab L4 GPU runtimes using greedy decoding (temperature=0.0, do_sample=False) for exact reproducibility.


Comparative Evaluation Summary Table

Category Probe ID & Task google/gemma-4-E4B-it openbmb/MiniCPM5-2B ibm-granite/granite-3.1-2b-instruct
Directional Accusative [1/20] "Cat runs into house" (Boundary entry) La kato iras al la domo (Fails -n case) Reasoning loop failure; questions if "run" = paroli La gato flugis al domo (Spanish leakage: gato, wrong verb)
[3/20] "She jumped onto table" Ŝi saltis sur la tablo (Fails -n case) Kiu jumis sur tavla (Hallucinated roots) Ela kantante al la tablo (Corrupted syntax)
[5/20] "Bird flew under bridge" La birdo flugis sub la ponton (Correct) La fonto fliĝis sub la pordo (Fabricated nouns) La avena flugas sub la brodo (Fabricated nouns)
Affix Derivation [6/20] Deconstruct malsanulejo Correctly breaks down mal-, san-, -ul-, -ej-, -o Claims -ulo = "child", -jo = particle Claims sanulejo derives from Latin ulcerare
[8/20] "Female teacher" derivation Fabricates virinisto (Fails -ino suffix rules) Claims feminine noun suffix is -a Fabricates docenteja (Spanish leakage)
[10/20] "Small place where bad people gather" Multi-word phrase: Malbona-kolekto Fabricates root malsamul Fabricates compound: mal-luko-jado-kompa
Correlative Matrix [11/20] Sequence: kiam, tiam, ĉiam, ___ Fabricates nuniam Fabricates neviĉ Claims ne is the correlative
[12/20] Correlative for "such / of that kind" Fabricates taŭa Fails matrix lookup; hallucinates kiel Claims de kio or kio
[13/20] Fill: Kiu -> Tiu -> Ĉiu -> ___ Neniu (Correct) Questions if words are Chinese/Esperanto Fabricates niunu
Community Idioms [16/20] Define krokodili "Acting overly dramatic / melodramatic" "Not a real Esperanto word" "Someone who is slow to learn"
[18/20] Define Volapūkaĵo "Not a standard Esperanto word" "Not a standard Esperanto word" "Language of the world" (Literal translation)
[20/20] Define samideano "Not a standard Esperanto word" "Not a standard Esperanto word" Claims it means "equal-to-all"

Detailed Failure Mode Taxonomy

  1. Subword Tokenization & Romance Vocabulary Leakage:
    • IBM Granite frequently spills over into Spanish, Italian, and Latin subwords (gato, corre, docente, skola, brodo), proving that its subword tokenizer merges Esperanto roots into Romance language frequency vectors.
  2. Correlative Table Lookup Collapse:
    • All three models fail to navigate the 5x9 correlative matrix. Both Gemma 4 and MiniCPM invent non-existent correlatives (nuniam, taŭa, neviĉ) rather than querying the rigid paradigm (neniam, tia).
  3. Morphological Agglutination Brittleness:
    • While Esperanto allows infinite valid compounding via affixes, models either invent false suffixes (Gemma's virinisto instead of instruistino) or completely fail to recognize foundational affixes like -ino (feminine) or -ej- (place).
  4. Complete Socio-Linguistic & Cultural Erasure:
    • Every tested model fails on community jargon. Terms like krokodili (speaking native language at an Esperanto event), Volapūkaĵo (gibberish/nonsense), and samideano (fellow Esperantist / "same-idea-member") are either declared "non-existent words" or hallucinated with false etymologies.

3. Proposed Technical Path Forward (Part 3)

To close this capability gap in sub-6B models without drastically expanding parameter count, I propose a 3-pillar intervention strategy:

  1. Targeted Data Curation & Synthetic Data Generation:
    • Collect parallel and monolingual corpora from authentic community archives (Tekstaro de Esperanto, Vikipedio, Monato).
    • Generate synthetic rule-based contrastive pairs explicitly probing the directional accusative (-n shift across all prepositions) and systematic correlative substitution matrices.
  2. Tokenizer Vocabulary Extension:
    • Add the 45 correlative words (Tabelvortoj) and standard 40+ Esperanto affixes as atomic tokens into the model's vocabulary, preventing subword fragmentation into Romance language roots.
  3. PEFT / QLoRA Alignment Tuning:
    • Apply QLoRA fine-tuning or Direct Preference Optimization (DPO) using preference pairs that explicitly penalize false derivations (e.g., rejecting virinisto or docenteja in favor of instruistino).

Repository Structure

├── README.md
├── esperanto_20_evaluation_probes.json
└── notebooks/
    ├── gemma4_e4b_esperanto_eval.ipynb
    ├── minicpm5_2b_esperanto_eval.ipynb
    └── granite_3.1_2b_esperanto_eval.ipynb
Total size
287 kB
Files
5
Last updated
Oct 22
Pre-warmed CDN
US EU US EU

Contributors