| --- |
| license: apache-2.0 |
| language: |
| - en |
| - it |
| tags: |
| - decision-model |
| - zero-shot-classification |
| - calibration |
| - distillation |
| - minijev |
| library_name: minijev |
| base_model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 |
| pipeline_tag: zero-shot-classification |
| --- |
| |
| # MiniJEV · General |
|
|
| A **small decision model** for English and Italian. You give it a **state** (optional |
| text), a **question**, and **any list of choices**. It returns the chosen option and a |
| **calibrated probability** for each choice. |
|
|
| It is not a chatbot and never generates text. It only scores the choices you give it, |
| so it cannot answer with something outside the list. This makes it fast, local and |
| predictable, and its confidence has a meaning. |
|
|
| ```python |
| dec.decide("", "Is a hotdog food?", ["yes", "no"]) # -> yes 0.79 |
| dec.decide("", "Is a tomato a fruit?", ["yes", "no"]) # -> yes 0.78 |
| dec.decide("chair", "What is this?", ["animal", "furniture", "vehicle"]) # -> furniture |
| dec.boolean("balena", "È un tipo di mammifero?", labels=("sì", "no")) |
| ``` |
|
|
| ## How it works |
|
|
| ``` |
| state ─┐ |
| question ─┼─ multilingual MiniLM-L12 (fine-tuned) ─► MiniJEV core ─► score per choice ─► softmax / T |
| choices ─┘ (each choice is encoded by meaning, so any list of 2..N choices works) |
| ``` |
|
|
| - **Text encoder**: `paraphrase-multilingual-MiniLM-L12-v2`, fine-tuned end to end. It has 117.7M parameters, 96M of which are the multilingual vocabulary. |
| - **MiniJEV core**: 0.83M parameters. It uses a bi-encoder scorer: each choice's meaning is compared with the state + question. |
| - **Choice handling**: choice order does not matter (the model is permutation-equivariant), and the number of choices is dynamic. |
| - **Calibration**: a temperature is fitted on validation data (T = 3.51). |
|
|
| This encoder was selected by an ablation study, summarised below. All variants were |
| trained on the same data and scored on validation, with unseen entities: |
|
|
| | variant | accuracy | |
| |---|---| |
| | **MiniLM, bi scorer, encoder fine-tuned** | **0.81** | |
| | MiniLM, bi, encoder frozen | 0.77 | |
| | MiniLM, cross-attention scorer, frozen | 0.74 | |
| | multilingual-e5-small, bi, fine-tuned | 0.74 | |
| | multilingual-e5-small, bi / cross, frozen | 0.54 / 0.50 | |
|
|
| On the earlier, smaller dataset, a fixed output head (which scores positions, not |
| the meaning of the choices) scored 0.37. It was not permutation-invariant. |
|
|
| ## Training data: distilled, not scraped |
|
|
| Qwen2.5-3B-Instruct was used **only offline, as a teacher**, to build the dataset. It |
| is never used at inference. |
|
|
| 1. The teacher listed examples for **100 categories**: food, animals, software, places, people, science and more. |
| 2. For each (entity, category) pair, the teacher's next-token probability of "yes" vs "no" was stored as a **soft label**. |
| 3. Entities the teacher itself did not place in their category were dropped. **2,221 entities** remained, giving about 24k decisions: yes/no questions and 2–10-way category choices, in English and Italian, with many phrasings. |
|
|
| Training used 50% hard labels and 50% KL distillation on the teacher distribution. |
| Splits are **by entity**: validation and test entities are never seen in training. |
|
|
| ## Results (unseen entities) |
|
|
| | test set | accuracy | ECE | |
| |---|---|---| |
| | teacher-labelled test (112 entities, 100 categories, up to 10 choices) | 0.81 | 0.055 | |
| | hand-built knowledge table (49 entities, 27 categories) | 0.83 | 0.058 | |
| | 30 hand-written questions (held-out entities and phrasings, it + en) | 23 / 30 | 0.13 | |
|
|
| **Latency on a laptop CPU**, one decision including the encoder: |
| - 1 thread: p50 31 ms, p95 41 ms, p99 45 ms |
| - 4 threads: p50 26 ms |
|
|
| ## Limits (read before using) |
|
|
| - **Knowledge is shallow.** It confuses close categories. For example, it answers |
| "is a whale a fish?" with yes, and it misplaces some less common technical names |
| (such as Redis). Treat it as a fast, calibrated first-pass classifier, not as a |
| knowledge base. |
| - **Italian is weaker than English.** |
| - **The dataset is synthetic** (template questions). Unusual phrasings work less well. |
| - **Use the confidence.** Low-confidence answers (under about 0.6) are right much |
| less often. Route them to a human or a larger model. |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://huggingface.co/markuz89/minijev-general && cd minijev-general/code |
| pip install -e ".[train,text]" |
| ``` |
|
|
| ```python |
| from minijev.inference.runtime import MiniJEV |
| |
| dec = MiniJEV.load("..") # the cloned repo folder (or the repo id) |
| d = dec.decide("", "Is Python a programming language?", ["yes", "no"]) |
| print(d.choice, d.confidence, d.scores) # calibrated distribution over your choices |
| print(d.raw_scores) # uncalibrated, for comparison |
| ``` |
|
|
| ## Files |
|
|
| - `model.safetensors`: MiniJEV weights |
| - `text_encoder/`: the fine-tuned MiniLM, with its tokenizer |
| - `config.json`: architecture, calibration and metrics |
| - `code/`: the `minijev` package |
|
|
| Everything is safetensors and JSON, with no pickle. |
|
|
