karpo-decision-31B / README.md
lionel-mp's picture
Copyright holder: MachinePulse Pte. Ltd.
b8e6fd7 verified
|
Raw History Blame Contribute Delete
10.4 kB
metadata
license: apache-2.0
library_name: transformers
base_model: google/gemma-4-31B-it
base_model_relation: finetune
pipeline_tag: text-classification
tags:
  - decision-model
  - decision-index
  - system-one
  - gemma4
model-index:
  - name: karpo-decision-31B
    results:
      - task:
          type: text-classification
          name: typed decision (choice / noul / score)
        dataset:
          type: mixed
          name: >-
            Decision Index 0.4.1, private tests (60% same skills, 40% new
            domains)
        metrics:
          - type: decision_index_full
            value: 73.86
            name: >-
              Decision Index 0.4.1, Full score (leaderboard, 12 November 2026;
              rank #1, the highest score)
        source:
          name: Decision Index leaderboard
          url: https://huggingface.co/spaces/multimodalart/jev-decision-index
      - task:
          type: text-classification
          name: typed decision (choice / noul / score)
        dataset:
          type: mixed
          name: >-
            Decision Index 0.3 public suite (140,178 scored requests; the index
            averages 37 benchmarks)
        metrics:
          - type: decision_index
            value: 75.05
            name: >-
              Decision Index 0.3, public index (chance-corrected; own run of the
              official kit, same value on the leaderboard)
        source:
          name: Own run of the official kit
          url: >-
            https://huggingface.co/datasets/MachinePulseAI/karpo-decision-31B-decision-index/tree/main/runs/karpo-decision-31B

Karpo Decision 31B

GitHub Decision Index: #1, Full 73.9 Results License: Apache 2.0

Ranked #1 on the Decision Index, with the highest Full score, 73.9, on the maintainers' private tests (edition 0.4.1, board of 12 November 2026, 300 entries).

Decision Index 0.4.1 leaderboard, Full score, 12 November 2026: karpo-decision-31B first with 73.9

Decision Index leaderboard, Full score, 12 November 2026.

A multimodal decision model: given a state (a web page and its screenshot, a document, a conversation: text, JSON and images) and typed questions about it, it returns a probability for every option of every question in one forward pass, without generating text. Fine-tuned from google/gemma-4-31B-it, with a linear readout over the options.

  • Parameters: 31.3B in bf16, plus a 255 × 5,376 readout in fp32
  • Input: multimodal, text and images (a screenshot at state.viewport_screenshot, or data URLs in images)
  • Questions: choice (up to 255 options), noul (true or false), score (a level on a scale)
  • Context: 32,768 tokens per sequence; a longer request is split by questions, each part with the full state
  • Interfaces: a Python API and an HTTP server in the System One format (POST /v1/systemone)
  • Licence: Apache-2.0

Results

Decision Index leaderboard

The board ranks by the Full score, which its maintainers measure on private tests only: 60% tests of the same skills as the public suite and 40% tasks from new domains.

Decision Index 0.4.1, board of 12 November 2026 karpo-decision-31B
Full score 73.9, rank #1, the highest of 300 entries
Private tests, same skills (on the public scale) 70.7
Private tasks, new domains (on the public scale) 78.6
Public index (our run, adopted by the board) 75.0
Median latency (one GPU, one request at a time) 94 ms per request
Calibration error 4.8 points

The board gives one rank to models within 0.9 points of the first model in a group: karpo-decision-31B (73.9) and GLiDE 27B (73.4) share #1.

Decision Index public suite

Our run of the official kit: 140,178 scored requests, 11 unsupported (a question over 32,768 tokens with its state). Results: runs/karpo-decision-31B.

score
Public index (chance-corrected) 75.0
Raw index 80.9
Breadth 74.7
Knowledge & Reasoning 65.5
Language Understanding 76.8
Retrieval & Classification 76.4
Tools & Automation 88.4
Arts & Human Taste 68.2

The Karpo Decision family

Score vs. size: Karpo Decision 0.8B to 31B on the Decision Index 0.3 public suite, with the other board models of each size class

Score vs. size on the Decision Index 0.3 public suite, board of 12 November 2026.

Model Base Parameters
karpo-decision-31B Gemma 4 31B 31.3B
karpo-decision-12B Gemma 4 12B 12.0B
karpo-decision-9B Qwen3.5 9B 8.4B
karpo-decision-4B Qwen3.5 4B 4.5B
karpo-decision-2B Qwen3.5 2B 2.2B
karpo-decision-0.8B Qwen3.5 0.8B 0.85B

How it works

  • A readout, not generation. Every option is labelled with an option code. At the Answer: ( slot that ends a question's branch, a linear readout scores 255 option codes and a softmax over the question's own options gives the probabilities. Nothing is decoded, so the answer is always one of the options.
  • Many questions, one forward pass. A request is one state and any number of questions, encoded as one sequence: the list of questions, the state, then one branch per question with its options.
  • Branch attention. A branch reads the shared block (questions and state) and itself, never another branch. The state is computed once per sequence, and no question sees another question's options.

The question-list-first layout follows FlexDM and jev-brain; the shared-block and branch rule follows Sieve-31B-v3.

Use

Inference code: github.com/machinepulse-ai/karpo-decision. Python 3.12 or later and one GPU; the weights take 62.5 GB, long requests more. Loading the checkpoint as a plain chat model does not reproduce its answers.

pip install "karpo-decision @ git+https://github.com/machinepulse-ai/karpo-decision"
from huggingface_hub import snapshot_download
from karpo_decision.decider import Decider

path = snapshot_download("MachinePulseAI/karpo-decision-31B")
decider = Decider.from_pretrained(path, device="cuda")
response = decider.decide({
    "state": "Order 1182: the customer asks to return a blender bought 45 days ago. Policy: returns within 30 days.",
    "questions": {
        "eligible": {"type": "noul", "instructions": "Is the return eligible under the policy?"},
        "action": {"type": "choice", "instructions": "What should the agent do?",
                   "criteria": {"refund": "Issue a full refund", "decline": "Politely decline the return",
                                "escalate": "Escalate to a supervisor"}},
    },
})
print(response["answers"])
# {"eligible": {"type": "noul", "noul": P(true)},
#  "action": {"type": "choice", "choice": <key>, "probabilities": {<key>: p, ...}, "confidence": c}}

HTTP server (POST /v1/systemone, GET /healthz):

CKPT=$(python -c 'from huggingface_hub import snapshot_download; print(snapshot_download("MachinePulseAI/karpo-decision-31B"))')
torchrun --standalone --nproc_per_node 1 -m karpo_decision.serve.server --checkpoint "$CKPT" --port 18440

Decision Index kit, installed separately:

pip install "decision-index @ git+https://github.com/apolinario/decision-index"
python -m decision_index run --engine karpo_decision.serve.engine:DecisionIndexEngine \
    --option checkpoint="$CKPT" --out runs/karpo-decision-31B

Lineage

google/gemma-4-31B → google/gemma-4-31B-it → karpo-decision-31B (fine-tuned)

Limitations

  • Not a chat or text-generation model; generation was not evaluated.
  • A question that does not fit in 32,768 tokens with its state is refused, never truncated.
  • It always picks one of the given options.

Licence

The weights and model files are licensed under the Apache License, Version 2.0. Copyright 2026 MachinePulse Pte. Ltd.

karpo-decision-31B is a modified version of google/gemma-4-31B-it (Apache-2.0): decoder weights fine-tuned, readout added. Gemma 4 stays under its own licence, a copy of which is in licenses/Gemma-4-LICENSE; see NOTICE.

The inference code (github.com/machinepulse-ai/karpo-decision) is licensed under Apache-2.0.

Attribution

Not required by the licence, but appreciated: if a product or service uses this model or a derivative, show

Powered by karpo-decision (MachinePulse)

with a link to https://huggingface.co/MachinePulseAI/karpo-decision-31B where the medium allows one.