| --- |
| library_name: transformers |
| license: gemma |
| base_model: google/gemma-4-12B |
| pipeline_tag: text-classification |
| tags: |
| - custom_code |
| - gemma4 |
| - typed-decisions |
| --- |
| |
| # System One Model (using Gemma 4 12B) |
|
|
| Using encoder and decoder from `google/gemma-4-12B` with its 262k-token |
| language-model head replaced by a low dimentional vector |
| that outputs one logit per answer. The forward pass gives the 26 logits. |
| Questions (`noul`, `choice`, `score`) are answered by softmax |
| over the logits for each question. |
|
|
| ## Benchmark on RTX 5090 |
|
|
| Median latency is **34.1ms** for a warm single-question decision. |
|
|
| ``` |
| Warm single-question decisions (latency_bench, 55 calls): |
| p50 34.1ms |
| p95 35.0ms |
| max 36.3ms |
| |
| Prefix cache (8 endings): |
| decisions changed: 0 of 8 |
| batched vs unbatched: 64.3ms vs 270ms |
| |
| 5-question request: 166ms |
| ``` |
|
|
| ## Comparison: Strands Decider 2B and the hosted Jev API |
|
|
| Same 39 labelled cases (15 `noul`, 12 `choice`, 12 `score`) sent through |
| `/v1/systemone` schema to each system. Local models ran on an Apple |
| Silicon Mac (64 GB, MPS, bf16) |
|
|
| | metric | this model (Gemma 4 12B, local) | [Strands Decider 2B v19](https://strandsagents.com/blog/introducing-strands-decider/) (local) | [TypeSafe Jev API](https://docs.typesafe.ai/introduction/quickstart) (hosted) | |
| |---|---|---|---| |
| | Mario | 5/5 | 5/5 | 5/5 | |
| | Tetris | 40/40 | 40/40 | 40/40 | |
| | `noul` accuracy | 15/15 | 15/15 | 15/15 | |
| | `noul` Brier (lower is better) | 0.000003 | 0.0338 | 0.00035 | |
| | `choice` accuracy | 11/12 | 12/12 | 12/12 | |
| | `score` accuracy | 11/12 | 5/12 | 12/12 | |
| | `score` mean error (levels) | 0.09 | 0.48 | 0.004 | |
| | 1-question p50 latency | 117 ms | 96 ms | 144 ms | |
| | 3-question p50 latency | 332 ms | 121 ms | 164 ms | |
| | Size | ~24 GB | ~2B params | n/a | |
|
|
| Test suite: 141 passed. Small hand-written sample |
|
|
| ## Doom played by the model. |
|
|
| <video controls width="640" src="https://huggingface.co/stephenlb/system-one-model/resolve/main/typesafe-replica-doom-game-only.mp4"></video> |
|
|
| ## Tetris played by the model. |
|
|
| <video controls width="640" src="https://huggingface.co/stephenlb/system-one-model/resolve/main/tetris.mp4"></video> |
|
|
| ## Super Mario Bros. World 1-1 played by the model. |
|
|
| <video controls width="640" src="https://huggingface.co/stephenlb/system-one-model/resolve/main/mario-world-1.mp4"></video> |
|
|
| ## Flappy Bird played by the model. |
|
|
| <video controls width="640" src="https://huggingface.co/stephenlb/system-one-model/resolve/main/jev-system-one-replica-flappy-bird.mp4"></video> |
|
|
| ## Model API |
|
|
| Requires `transformers>=5.17`. The repo ships custom code, so pass |
| `trust_remote_code=True`. |
|
|
| ```python |
| from transformers import AutoModelForMultimodalLM, AutoTokenizer |
| |
| repo = "stephenlb/system-one-model" |
| tokenizer = AutoTokenizer.from_pretrained(repo) |
| model = AutoModelForMultimodalLM.from_pretrained( |
| repo, trust_remote_code=True, dtype="bfloat16", device_map="auto" |
| ) |
| |
| result = model.system_one( |
| tokenizer, |
| state="I was charged twice for order A-104.", |
| questions={ |
| "refund": {"type": "noul", "instructions": "Does the text request a refund?"}, |
| "team": { |
| "type": "choice", |
| "instructions": "Which team should handle this?", |
| "criteria": {"billing": "Charges", "returns": "Refunds"}, |
| }, |
| }, |
| ) |
| print(result["answers"]) |
| ``` |
|
|
| ## Pipeline API |
|
|
| ```python |
| from transformers import pipeline |
| |
| pipe = pipeline("system-one", model=repo, trust_remote_code=True, dtype="bfloat16", device_map="auto") |
| print(pipe({"state": "I was charged twice.", "questions": {...}})) |
| ``` |
|
|
| ## Output |
|
|
| ```json |
| {"answers": { |
| "refund": {"type": "noul", "noul": 0.93}, |
| "team": {"type": "choice", "choice": "billing", |
| "probabilities": {"billing": 0.97, "returns": 0.03}, "confidence": 0.8}, |
| "urgency": {"type": "score", "score": 3.1, "legend": {"0": "..."}, |
| "probabilities": {"0": 0.0, "1": 0.1}, "confidence": 0.5}}} |
| ``` |
|
|
| `noul` is the probability of yes. `choice` returns the top label with |
| probabilities per option. `score` returns the expected level over an ordered |
| `criteria` list. Pass `temperature=` (default 0.7) to sharpen or soften. |
|
|
| ## Examples |
|
|
| Outputs below are real results from this model (pipeline defaults, temperature 0.7). |
|
|
| ### `noul`: yes/no probability |
|
|
| `noul` is the probability the answer is yes. Ask several yes/no questions about |
| one text in a single call. |
|
|
| ```python |
| pipe({ |
| "state": "Please cancel my subscription and refund this month's charge.", |
| "questions": { |
| "refund": {"type": "noul", "instructions": "Does the text request a refund?"}, |
| "cancel": {"type": "noul", "instructions": "Does the text ask to cancel a subscription?"}, |
| "sports": {"type": "noul", "instructions": "Does the text talk about sports?"}, |
| }, |
| })["answers"] |
| # {"refund": {"type": "noul", "noul": 0.9996}, |
| # "cancel": {"type": "noul", "noul": 0.9998}, |
| # "sports": {"type": "noul", "noul": 0.0023}} |
| ``` |
|
|
| Optional `criteria` overrides the yes/no wording: `{"true": "...", "false": "..."}`. |
|
|
| Another text, different questions: tone and content checks. |
|
|
| ```python |
| pipe({ |
| "state": "Thanks so much for the quick help, the new dashboard looks great!", |
| "questions": { |
| "polite": {"type": "noul", "instructions": "Is the text polite?"}, |
| "complaint": {"type": "noul", "instructions": "Does the text contain a complaint?"}, |
| "question": {"type": "noul", "instructions": "Does the text ask a question?"}, |
| }, |
| })["answers"] |
| # {"polite": {"type": "noul", "noul": 0.9991}, |
| # "complaint": {"type": "noul", "noul": 0.0009}, |
| # "question": {"type": "noul", "noul": 0.0080}} |
| ``` |
|
|
| ### `choice`: pick one option |
|
|
| `criteria` maps each option label to a description. You get the top label, |
| a probability for every option, and a `confidence` (1.0 means all mass on one |
| option, 0.0 means uniform). Up to 26 options. |
|
|
| ```python |
| pipe({ |
| "state": "My package says delivered but nothing arrived at my door.", |
| "questions": { |
| "team": { |
| "type": "choice", |
| "instructions": "Which team should handle this?", |
| "criteria": { |
| "billing": "Charges, invoices, payment problems", |
| "shipping": "Delivery status, delays, lost packages", |
| "returns": "Exchanges, refunds, wrong or damaged items", |
| }, |
| }, |
| }, |
| })["answers"] |
| # {"team": {"type": "choice", "choice": "shipping", |
| # "probabilities": {"shipping": 0.9977, "returns": 0.0016, "billing": 0.0007}, |
| # "confidence": 0.984}} |
| ``` |
|
|
| Intent classification with four options: |
|
|
| ```python |
| pipe({ |
| "state": "Can you add a dark mode to the mobile app? My eyes hurt at night.", |
| "questions": { |
| "intent": { |
| "type": "choice", |
| "instructions": "What is the intent of the message?", |
| "criteria": { |
| "bug_report": "Something is broken or not working as expected", |
| "feature_request": "Asks for new functionality or an improvement", |
| "praise": "Compliments or thanks", |
| "question": "Asks how to do something", |
| }, |
| }, |
| }, |
| })["answers"] |
| # {"intent": {"type": "choice", "choice": "feature_request", |
| # "probabilities": {"feature_request": 0.9781, "bug_report": 0.0094, |
| # "question": 0.0079, "praise": 0.0046}, |
| # "confidence": 0.907}} |
| ``` |
|
|
| ### `score`: ordered scale |
|
|
| `criteria` is an ordered list of levels (lowest first, 2 to 26 levels). `score` |
| is the expected level `sum(i * p_i)`, so 1.94 below means mostly level 2 with a |
| little level 1. `legend` maps each level to its description. |
|
|
| ```python |
| pipe({ |
| "state": "The checkout page crashes every time I press Pay and I can't buy anything.", |
| "questions": { |
| "severity": { |
| "type": "score", |
| "instructions": "How severe is the reported issue?", |
| "criteria": [ |
| "Cosmetic; no impact to functionality", |
| "Broken or degraded feature, but workaround exists", |
| "Blocking issue; no workaround exists", |
| ], |
| }, |
| }, |
| })["answers"]["severity"] |
| # {"type": "score", "score": 1.942, |
| # "legend": {"0": "Cosmetic; ...", "1": "Broken or degraded ...", "2": "Blocking issue; ..."}, |
| # "probabilities": {"0": 0.0018, "1": 0.0542, "2": 0.9440}, "confidence": 0.796} |
| ``` |
|
|
| A five-point scale on mixed feedback. The distribution is spread out, so |
| `score` lands between levels and `confidence` is low: |
|
|
| ```python |
| pipe({ |
| "state": "The delivery was two days late, the box was dented, but the product itself works fine.", |
| "questions": { |
| "satisfaction": { |
| "type": "score", |
| "instructions": "How satisfied is the customer overall?", |
| "criteria": ["Very dissatisfied", "Dissatisfied", "Neutral", "Satisfied", "Very satisfied"], |
| }, |
| }, |
| })["answers"]["satisfaction"] |
| # {"type": "score", "score": 1.408, |
| # "legend": {"0": "Very dissatisfied", "1": "Dissatisfied", "2": "Neutral", |
| # "3": "Satisfied", "4": "Very satisfied"}, |
| # "probabilities": {"0": 0.1144, "1": 0.5705, "2": 0.1635, "3": 0.0957, "4": 0.0560}, |
| # "confidence": 0.223} |
| ``` |
|
|
| ### Mixing types in one call |
|
|
| All questions in a call see the same `state` and run in one batched forward |
| pass. `state` may also be a dict or list; it is serialized to JSON. |
|
|
| ```python |
| pipe({ |
| "state": {"customer": "A-104", "message": "The checkout page crashes when I press Pay."}, |
| "questions": { |
| "urgent": {"type": "noul", "instructions": "Does the text express urgency?"}, |
| "sentiment": { |
| "type": "choice", |
| "instructions": "What is the sentiment of the text?", |
| "criteria": { |
| "positive": "Happy or satisfied", |
| "neutral": "Factual, no strong emotion", |
| "negative": "Unhappy or frustrated", |
| }, |
| }, |
| "severity": { |
| "type": "score", |
| "instructions": "How severe is the reported issue?", |
| "criteria": ["Cosmetic", "Degraded, workaround exists", "Blocking"], |
| }, |
| }, |
| }, temperature=0.5, include_letter_logits=True) |
| ``` |
|
|
| `temperature` below 0.7 sharpens the probabilities; above it, softens them. It |
| never changes which option wins. `include_letter_logits=True` adds the raw A-Z |
| logits per question under `letter_logits`. |
|
|
| ## Raw logits |
|
|
| ```python |
| enc = tokenizer(prompts, return_tensors="pt", padding=True) # padding_side="left" |
| logits = model.letter_logits(enc.input_ids, enc.attention_mask) # [batch, 26] |
| ``` |
|
|
| # Blocks.ai |
|
|
| We needed an open weight model that offered the capabilities of Jev System One Model. |
| Most common AI Agents require decisions making. |
| The System One model approach is a great new way to do this. |
| [Blocks.ai](https://blocks.ai) is the secure network for the Internet of Agents (IoA). |
| Whether connecting agents to users in your organization or making |
| your agents available for public use, |
| Blocks.ai makes your agents securely discoverable and callable |
| by everyone who needs them most. |
| [We Rebuilt Jev's API on an Open Model and Used It to Play Doom](https://blocks.ai/blog/jev-open-model-doom). |
| TypeSafe AI's Jev turns unstructured input into typed decisions and probabilities. |
| We rebuilt the API with an open model, reached 113ms median latency on an M4 Mac, and used it to play Doom. |
|
|
| # Github Source System One Model |
|
|
| https://github.com/stephenlb/system-one-model |
|
|
