| ---
|
| license: apache-2.0
|
| base_model: jhu-clsp/ettin-encoder-400m
|
| library_name: onnxruntime
|
| pipeline_tag: text-classification
|
| tags: [decision-model, typed-decisions, calibrated, cross-encoder, on-device, onnx, aubin, norovox, preview]
|
| ---
|
|
|
| 
|
|
|
| # AUBIN Mini — AUBIN by Norovox
|
|
|
| > **Release** · beats Kev on its own locked suites (trained sources .852 vs .838, new sources .737 vs .697) while running on a phone CPU.
|
|
|
| **Typed decisions on the phone: question + options + state in, calibrated probabilities out. CPU only, 267 MB.**
|
|
|
| AUBIN Mini is the on-device member of the AUBIN family. It answers structured questions (choice, yes / no, any
|
| number of options) about a piece of state (a ticket, a message, a form, a review) and returns a probability for every
|
| option, so an app can act on the answer or ask for help when the confidence is low. It needs ONNX Runtime and a
|
| tokenizer, nothing else: no torch, no GPU, no network after the first download.
|
|
|
| | | |
|
| |---|---|
|
| | parameters | 395M (ModernBERT-style encoder, inputs up to 512 tokens) |
|
| | phone file | `aubin_mini_nb4_g8.onnx`, **267 MB** (4-bit block weights, 8-bit embeddings), 5.9x smaller than fp32 |
|
| | also included | `aubin_mini_nb4.onnx` (422 MB), `model.safetensors` (full-precision PyTorch weights for fine-tuning) |
|
| | runtime | ONNX Runtime CPU (desktop, server, Android / iOS builds of ONNX Runtime) |
|
| | output | one calibrated probability per option (temperature fitted on Kev's development split) |
|
| | languages | trained and measured in English |
|
|
|
| ## Results on Kev's public locked suites
|
|
|
| Same items and metrics as [jaredpalmer/kev](https://github.com/jaredpalmer/kev) (`decision-v7` test = 1,440 questions
|
| from sources Kev trained on; `transfer-v4` test = 764 records from sources it never trained on). Kev numbers are the
|
| published ones from Kev's README. Lower Brier / ECE is better.
|
|
|
| | model | file | trained sources acc (test) | new sources acc (test) | trained sources Brier / ECE (test) | new sources Brier (test) |
|
| |---|---|---|---|---|---|
|
| | **AUBIN Mini, phone build** (`aubin_mini_nb4_g8.onnx`) | **267 MB** | **0.847** | 0.622 | 0.222 / 0.039 | 0.503 |
|
| | AUBIN Mini, 4-bit (`aubin_mini_nb4.onnx`) | 422 MB | 0.846 | 0.617 | 0.223 / 0.032 | 0.502 |
|
| | AUBIN Mini, full precision (PyTorch) | 1.58 GB | 0.844 | 0.618 | 0.226 / 0.036 | 0.503 |
|
| | EmbeddingGemma-300M, same training recipe (ours) | — | 0.763 | 0.487 | 0.330 / 0.053 | 0.613 |
|
| | Kev-0.8B (Qwen3.5-0.8B + LoRA) | — | 0.838 | 0.697 | — | 0.416 |
|
| | Kev-4B (Qwen3.5-4B + LoRA) | — | 0.865 | 0.838 | — | 0.242 |
|
|
|
| The phone build was measured with this repository's own code (`code/eval_kev.py`, ONNX Runtime CPU) on all 1,440 +
|
| 764 test questions; its tokenizer reproduces the training tokenizer exactly (960 / 960 checked pairs).
|
| On sources it was trained on, the 267 MB phone build is ahead of Kev-0.8B; on sources never seen in training,
|
| Kev-0.8B and larger generative models are ahead. EmbeddingGemma-300M row: Google's embedding model trained by us with
|
| exactly the same data, teacher and number of steps, as a like-for-like on-device baseline.
|
|
|
| Speed (one question = all of its options): on an ordinary Windows PC CPU (all cores) a 2–3 option decision took
|
| 0.6–1.0 s and a two-question request 2.6 s, after a one-time load. Single-thread phone stand-in: median **5.9 s per question on one CPU thread** of a 4-core Kaggle CPU
|
| machine with four evaluation processes sharing it; 2.9 s with a single process (300 development questions).
|
|
|
| Per source, phone build, `decision-v7` test (sources seen in training):
|
|
|
| | source | n | accuracy |
|
| |---|---|---|
|
| | agnews | 116 | 0.905 |
|
| | agnews_yn | 184 | 0.908 |
|
| | amazon | 80 | 0.588 |
|
| | banking77 (77 options) | 116 | 0.724 |
|
| | boolq | 80 | 0.750 |
|
| | contrastive_age_eligibility | 40 | 1.000 |
|
| | contrastive_quantity_limit | 40 | 1.000 |
|
| | contrastive_return_window | 40 | 0.775 |
|
| | contrastive_spend_threshold | 76 | 1.000 |
|
| | dbpedia14 | 116 | 0.974 |
|
| | imdb | 80 | 0.950 |
|
| | mnli | 116 | 0.819 |
|
| | sst5 | 80 | 0.588 |
|
| | trec | 116 | 0.966 |
|
| | yelp | 80 | 0.637 |
|
| | yelp_yn | 80 | 0.938 |
|
|
|
| Per source, phone build, `transfer-v4` test (sources never seen in training):
|
|
|
| | source | n | accuracy |
|
| |---|---|---|
|
| | composition_final_combination | 32 | 0.500 |
|
| | composition_final_exception | 32 | 0.812 |
|
| | composition_final_negation | 32 | 0.750 |
|
| | contrastive_authorization | 40 | 0.850 |
|
| | contrastive_deadline | 40 | 0.450 |
|
| | emotion | 116 | 0.491 |
|
| | mmlu | 116 | 0.345 |
|
| | paws | 80 | 0.550 |
|
| | qnli | 80 | 0.787 |
|
| | sciq | 116 | 0.784 |
|
| | tweet_offensive | 80 | 0.775 |
|
|
|
| ## Use
|
|
|
| ```bash
|
| pip install "git+https://huggingface.co/emrevrg/AUBIN-Mini#subdirectory=code"
|
| ```
|
|
|
| ```python
|
| from aubin_mini import AubinMini
|
|
|
| m = AubinMini.from_pretrained("emrevrg/AUBIN-Mini") # phone build (267 MB); variant="nb4" for the 422 MB file
|
| m.decide({"ticket": "Package arrived broken, customer wants money back"},
|
| {"route": {"type": "choice", "instructions": "Which team should handle this?",
|
| "criteria": {"billing": "refunds and charges", "shipping": "delivery and damage", "tech": "software"}},
|
| "urgent": {"type": "noul", "instructions": "Is this urgent?"}})
|
| # → {"route": {"answer": "...", "confidence": ..., "probabilities": {"billing": ..., "shipping": ..., "tech": ...}},
|
| # "urgent": {"answer": True/False, "confidence": ..., "probabilities": {"false": ..., "true": ...}}}
|
|
|
| m.choose("Which team should handle this?", ["billing", "shipping", "technical support"],
|
| state="The app crashes every time I open the camera screen.") # → {"answer": "technical support", ...}
|
| ```
|
|
|
| Command line and a local HTTP endpoint:
|
|
|
| ```bash
|
| aubin-mini decide request.json # {"state": ..., "questions": {...}}
|
| aubin-mini serve --port 8009 # POST /decide (same JSON), GET /health
|
| ```
|
|
|
| Reproduce the table: `python code/eval_kev.py --model emrevrg/AUBIN-Mini --variant phone` (add `--shard k/n` to split).
|
|
|
| ## How it works
|
|
|
| * **Cross-encoder, one score per option.** Each option is read together with the question and the state
|
| (`QUESTION: … OPTION: …` ‖ `STATE: …`), the encoder's first token is mapped to a single score, and the scores of a
|
| question are turned into probabilities with a softmax at the fitted temperature. Because options are scored one by
|
| one, there is no limit on the number of options (banking77 has 77) and option order does not matter.
|
| * **Long state is cut, the question never is**: the state side is truncated at 512 tokens.
|
| * **Training**: Ettin-encoder-400M, fine-tuned on Kev's `decision-v7` train split with knowledge distillation from the
|
| AUBIN ensemble's probabilities (α 0.7) plus the gold label; checkpoint chosen on the development split only.
|
| No transfer source was used for training, selection or calibration.
|
| * **Phone build**: ONNX export, 4-bit block-wise (32) symmetric weight quantization of every matrix multiply and 8-bit
|
| embedding table. On the same 300 development questions it scores 0.833 vs 0.837 for fp32 (one question).
|
|
|
| ## Notes
|
|
|
| * Every number comes from the scripts in `code/` on Kev's public suites; nothing is hand-edited.
|
| * Latency is measured on one CPU thread on a Kaggle CPU machine, as a stand-in for a phone core, not on a phone.
|
| ONNX Runtime uses all cores by default, which is faster.
|
| * The phone file uses ONNX Runtime's `MatMulNBits` operator. The standard `onnxruntime` pip package includes it; on
|
| mobile use the full ONNX Runtime package, not a reduced-operator build.
|
| * Base model: [jhu-clsp/ettin-encoder-400m](https://huggingface.co/jhu-clsp/ettin-encoder-400m) (MIT). This release:
|
| Apache-2.0.
|
|
|
| AUBIN by Norovox · version 0.1.0
|
|
|
| ---
|
|
|
| ### Support Norovox
|
| Norovox is built by **one 17-year-old high school student in Türkiye**: no company, no sponsor, no budget.
|
| Every model on this page was trained on free GPU quotas that run out within days. When they run out, training stops
|
| halfway and the next, stronger version has to wait, sometimes for a week.
|
|
|
| **Your support goes directly into GPU time.** Even a small contribution lets a training run finish instead of stopping,
|
| and helps the Nythos, N1, AUBIN and Norovox Embed models get better faster. Supporters are credited in the model cards
|
| and get early access to new releases.
|
|
|
| **Contact:** zgremre@gmail.com · emrevrgdev@gmail.com
|
|
|
| <sub>🇹🇷 Norovox'u Türkiye'de 17 yaşında bir lise öğrencisi tek başına geliştiriyor: şirket yok, sponsor yok, bütçe yok. Buradaki modellerin hepsi birkaç günde biten ücretsiz GPU kotalarıyla eğitildi; kota bitince eğitim yarıda kalıyor ve daha güçlü sürüm bekliyor. Desteğin doğrudan GPU süresine gider; küçük bir katkı bile bir eğitimin yarıda kalmadan tamamlanmasını sağlar. Destekçiler model kartlarında anılır ve yeni sürümlere erken erişir. İletişim: zgremre@gmail.com · emrevrgdev@gmail.com</sub>
|
|
|
