Spaces:
Running
Running
File size: 5,434 Bytes
4905d8d 7c9e8d9 4905d8d 7c9e8d9 4905d8d 7c9e8d9 4905d8d 7c9e8d9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | ---
title: ModelOps Research Lab
emoji: "🧪"
colorFrom: indigo
colorTo: blue
sdk: static
app_file: index.html
pinned: false
license: mit
short_description: Reproducible multilingual NLP model evaluation and audits.
tags:
- static
- model-evaluation
- nlp
- multilingual
- reproducibility
- experiment-tracking
- explainable-ai
---
# 🧪 ModelOps Research Lab v2.0
An offline-first, zero-cost browser application for **reproducible multilingual text-classification experiments**, created for the applied AI research portfolio of Rodrigo Pantoja (`@Hugdrigo`).
## Purpose
Move beyond attractive demos to **measured model behavior**: training, fixed evaluation partitions, experimental comparisons, error inspection, language/domain slices, bootstrap intervals, source audits, and traceable result exports.
## Quickstart
1. Open **App**. The four transparent lexical baselines are fitted *in your browser* on the included 842 synthetic training examples.
2. See **Overview** for the 54-item novel-pattern challenge, and switch to the controlled 130-item IID test.
3. Open **Experiments** and click **Run four baselines** to create a locally saved run with seed-controlled bootstrap sampling.
4. Visit **Slice analysis**, **Error lab**, **Shift diagnostics** and **Data audit**.
5. Export predictions (CSV), metrics (JSON), reports (Markdown), and run history (JSON).
## Exact research data and licensing
Source: **Academic Rhetorical Intent Mini v1.1** (originally developed as an earlier portfolio research dataset). All **1,026 examples are synthetic-curated**, written for educational experimental use; no real scholarly manuscripts or student submissions were copied. Distribution: 842 training, 130 controlled IID, 54 held-out novel-pattern challenge. Languages: English, Spanish, Portuguese. Six rhetorical labels: background, objective, method, result, limitation, implication.
**Dataset**: CC BY 4.0. See `DATA_LICENSE.md` and `ORIGINAL_DATASET_CARD.md`. **App and baseline engine**: MIT. See `LICENSE`. The README's `license: mit` denotes the Space's application code and does not override the included dataset attribution.
## Tested baselines and evaluation
All four algorithms are implemented in `engine.js` and trained from `train.json` on every page load:
| Method | Controlled IID Accuracy (130) | Novel-pattern Accuracy (54) | Novel-pattern Macro F1 |
|---|---:|---:|---:|
| Multinomial Naive Bayes | 100.0% | 74.1% | 73.6% |
| Word TF-IDF centroid | 98.5% | 72.2% | 70.6% |
| Character trigram TF-IDF centroid | 95.4% | 70.4% | 68.4% |
| Weighted word/character hybrid | 97.7% | **77.8%** | 76.2% |
These are **new browser baselines**; they do not reproduce the previously trained Python model. The hybrid is a weighted combination of two baselines, not a separately trained neural network. Scores are derived from the fixed **synthetic** dataset, not a scientific claim of transferable academic performance. See `MODEL_CARD.md`, `METHODS.md` and `LIMITATIONS.md`.
A historical *different Python model* reported 79.6% challenge Accuracy and only 35.7% on a very small 28-sentence PubMed development smoke test. The latter is shown separately in the dashboard to remind users about domain shift, **not** to imply that the browser classifiers were evaluated on PubMed.
## Reproduce without a browser
```bash
node --test test_engine.js
node benchmark_cli.js
```
Node 18+ required. No npm install needed. In-browser operation needs JavaScript and no paid API. These tests do not require internet.
## Methodological notes
- No training or hyperparameter fitting uses the holdout splits.
- Method weights and feature definitions are specified in code, **not optimized on the challenge**.
- Macro F1 averages labels present in the evaluated group. For 130/54 full sets, all six labels are present.
- Bootstrap intervals are **resampling intervals across fixed held-out rows**. They are not estimates of uncertainty due to training sets, label noise or real-world deployment.
- The Data audit checks exact text / ID overlap. **Semantic or template-level leakage is not excluded.**
- Scores from synthetic examples must **never** be used for grading, employment, admissions or certification of educational quality.
## Repository structure
- `index.html`, `styles.css`, `app.js`, `engine.js`: standalone Static Space app and model code.
- `train.json`, `iid_test.json`, `challenge.json`: original fixed splits used by the live app.
- `academic_rhetoric_intent.csv`: original combined synthetic dataset, with source metadata.
- `dataset_manifest.json`, `DATA_CARD.md`, `MODEL_CARD.md`: provenance and technical cards.
- `evaluation_results.json`: independently recomputed baseline metrics.
- `test_engine.js`, `benchmark_cli.js`, `test_browser.py`: reproducibility and browser QA.
- `DEPLOYMENT_ES.md`: easy Spanish uploading guide.
- Other MD files: limitations, methods, source license and release checks.
## Privacy, cost and limitations
The live app sends **no user sentence** to any server. Model training and scoring operate locally in the browser. Short experiment-summary history uses browser local storage; exports are local downloads. The bundled texts are synthetic and public. Zero paid model inference or managed compute is required.
See `SECURITY.md` and `LIMITATIONS.md` for caveats. Please treat the application as a **research demonstration**, not as a grading or decision-making system.
|