| --- |
| title: Multilingual ModelBench |
| emoji: 📊 |
| colorFrom: blue |
| colorTo: indigo |
| sdk: static |
| app_file: index.html |
| pinned: false |
| license: mit |
| short_description: Reproducible multilingual information retrieval evaluation. |
| tags: |
| - information-retrieval |
| - multilingual |
| - nlp |
| - benchmarking |
| - research |
| - evaluation |
| --- |
| |
| # Multilingual ModelBench | Rodrigo Pantoja |
|
|
| An interactive, reproducible evaluation dashboard for **multilingual information retrieval** in English, Spanish and Portuguese. Portfolio release **v3.0**. |
|
|
| ## Live application |
|
|
| The static dashboard is fully browser-based. It loads the committed, locally executed evaluation outputs from `benchmark-data.json`, with no login, token, API or server compute needed. |
|
|
| **What you can do** |
|
|
| 1. Compare four measured lexical retrieval methods using Accuracy@1 and Mean Reciprocal Rank (MRR). |
| 2. Switch between a 30-case monolingual split and an 18-case cross-language split. |
| 3. Inspect segment performance by language, domain, and language direction. |
| 4. Inspect every original evaluation question, its three candidate passages, and the per-method ranking and raw scores. |
| 5. Export the benchmark summary as a CSV file. |
| 6. Review registered but **not yet evaluated** semantic models without claiming invented scores. |
|
|
| ## Dataset provenance |
|
|
| The included **Research Retrieval Eval Mini** benchmark contains **48 curated evaluation questions**, not an independently sampled production test set. Each query contains exactly one designated positive passage, one hard negative and one easy negative. The corpus covers academic research, applied AI, education, business, and Latin America in English, Spanish, and Portuguese. |
|
|
| The `benchmark-data.json` file contains the complete test cases, recorded result summaries from the earlier Python runs, and the per-query rankings regenerated with scikit-learn. It is bundled for transparency and offline reproducibility. |
|
|
| **Data license: CC BY 4.0.** The dataset contents in `benchmark-data.json` are *not* MIT-licensed. Code and user interface: MIT. |
|
|
| ## Methods and metrics |
|
|
| | Baseline | Measured | Method | |
| |---|---|---| |
| | Character TF-IDF | Yes | Character-boundary n-grams, length 3 to 5 | |
| | Word TF-IDF | Yes | Word n-grams, length 1 to 2 | |
| | Hybrid TF-IDF | Yes | Weighted blend: 0.6 character + 0.4 word | |
| | BM25 | Yes | Probabilistic term frequency scoring | |
| | `intfloat/multilingual-e5-small` | No | Configured semantic candidate | |
| | `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` | No | Configured semantic candidate | |
| | `sentence-transformers/LaBSE` | No | Configured semantic candidate | |
|
|
| **Accuracy@1:** proportion of queries with the positive passage in rank 1. |
|
|
| **MRR:** average inverse rank of the positive passage (1 / rank). |
|
|
| ### Stored results |
|
|
| | Split | Character TF-IDF Accuracy@1 | Word TF-IDF Accuracy@1 | Hybrid Accuracy@1 | BM25 Accuracy@1 | |
| |---|---:|---:|---:|---:| |
| | Monolingual, 30 queries | 96.7% | 93.3% | 96.7% | 90.0% | |
| | Cross-language, 18 queries | 94.4% | 27.8% | 94.4% | 27.8% | |
|
|
| These are **small-sample diagnostics**, not generalized accuracy claims. A high cross-language character score can reflect cognates, topic overlap, or evaluation-set design and does not guarantee transfer to unrelated text. |
|
|
| ## Reproduce the benchmark locally |
|
|
| The eight uploaded files include `benchmark.py`, which uses the identical scoring logic as the original research prototype, adapted to read the included `benchmark-data.json`. |
|
|
| ```bash |
| pip install -r requirements-benchmark.txt |
| python benchmark.py --models lexical --split monolingual |
| python benchmark.py --models lexical --split cross-language |
| ``` |
|
|
| To attempt neural models, first install `sentence-transformers` in a Python environment with Internet access and adequate resources: |
|
|
| ```bash |
| pip install sentence-transformers |
| python benchmark.py --models all --split cross-language |
| ``` |
|
|
| Outputs go to `my-results-monolingual.json` or `my-results-cross-language.json`, unless `--output` is specified. The public dashboard only displays committed `benchmark-data.json` until it is intentionally updated. It does **not** silently run the neural models or publish local result files. |
|
|
| ## Limitations, ethics and future work |
|
|
| - 48 curated query cases do not estimate performance on general unseen documents or every language variant. |
| - Each query has three fixed candidate passages, not a corpus-scale retrieval task. |
| - Lexical overlap between related languages can inflate performance of character-level methods. |
| - The original evaluation contained no measured neural-embedding results. None are fabricated. |
| - Average latency in stored historical outputs depends on the execution environment and should not be compared to benchmarks from other hardware. |
| - More difficult paraphrases, independent annotation and larger held-out datasets are essential for a stronger claim. |
| - The benchmark contains no private personal data or user CVs. |
|
|
| **Next milestone:** run and evaluate three multilingual embedding models against the same examples, then add an external validation set, significance analysis and broader topical coverage. |
|
|
| ## File guide |
|
|
| - `index.html` : main Space page and dashboard UI. |
| - `styles.css` : responsive design. |
| - `app.js` : interactivity and display logic. |
| - `benchmark-data.json` : complete test cases, observed results and case-level evidence. |
| - `benchmark.py` : reproducible offline benchmarking, with optional semantic models. |
| - `requirements-benchmark.txt` : Python reproducibility dependencies. |
| - `LICENSE` : MIT software license with data license distinction. |
| - `README.md` : Space metadata, methodology and attribution. |
|
|
| Author: **Rodrigo Pantoja**. Part of an open-source applied AI, data and research tooling portfolio. |
|
|
