Arbiter-8B (v0.5)

Arbiter-8B is a Jev-inspired decision model built on Alibaba's Qwen3-8B, for wealth and asset management. Arbiter is aimed to be run on-prem at low cost. It reads financial text (news articles, headlines, SEC Form 10-K passages, central-bank statements) and answers a typed question about it, such as the type of corporate event, the implied outlook for a company, the sentiment toward it, or the policy stance of a sentence. It returns a calibrated probability for every option in a single forward pass, with no generated text.

This repository holds Arbiter v0.5. It replaces v0.4, which is still available in the repository history.

Results at a glance

Same items, same question and same option definitions for every system. EDT and FOMC are scored on their held-out test splits; SEntFiN was never trained on.

Benchmark	Arbiter v0.5	Qwen3-8B (base)	Jev 1.14	OpenAI Decisions API
EDT, corporate event detection, 1,933 articles, accuracy	94.2%	89.3%	88.3%	90.8%
FOMC, policy stance, 496 sentences, accuracy	67.5%	55.0%	66.7%	67.3%
SEntFiN, entity sentiment, 1,000-headline sample, accuracy	75.7%	75.4%	81.6%	78.0%
SEntFiN, signals caught (positive/negative labels called correctly)	92.0%	82.8%	91.0%	91.7%
SEntFiN, signal precision (positive/negative calls that are right)	72.2%	76.3%	78.8%	74.8%

• EDT: Arbiter leads all three systems; every gap is significant (paired McNemar, z ≥ 7.5).
• FOMC: Arbiter, Jev and OpenAI are level (z ≤ 0.4); all three are far ahead of the base model (z 4.8).
• SEntFiN: Arbiter catches the most positive and negative signals but is less precise than Jev and OpenAI; it labels too few headlines neutral.

What changed from v0.4

v0.5 was trained to follow the option definitions it is given, to separate genuine corporate events from look-alikes, and to read central-bank language.

	v0.4	v0.5
EDT accuracy	83.5%	94.2%
FOMC accuracy	60.1%	67.5%
SEntFiN accuracy (full benchmark, 14,329 headlines, zero-shot)	76.0%	76.4%
EDT new_contract precision	38.3%	86.7%
EDT none recall	81.2%	95.2%
Answers that change when options are shuffled, FOMC	15.5%	4.8%
Calibration error (ECE), EDT / FOMC	0.099 / 0.127	0.012 / 0.067
Human-labelled dev set (151, news + 10-K)	85.4%	84.1%
Human-labelled test set (56, 10-K)	83.9%	82.1%

The two human-labelled sets moved by one or two items, which is within noise: the new data neither helped nor hurt the original news and 10-K tasks. One clear regression: EDT clinical-trial recall fell from 87.1% to 54.8% (precision held at 79.1%); missed trials mostly go to none.

Model details

• Model type: decision model (single-pass scoring of a fixed set of options), built on the Qwen3 causal language model (Qwen3ForCausalLM, 8B parameters)
• Base model: Qwen/Qwen3-8B, fine-tuned with LoRA (rank 32) and a decision head
• Language: English
• Domain: financial news, headlines, SEC Form 10-K filings and central-bank communication, for wealth and asset management
• Precision: fp16 (the fine-tuning is merged into the weights)
• Method: follows "Releasing the Inner Jev" (Geuter et al., 2026)

How it works

You give Arbiter a text, a question and a set of options, each with a short definition. It scores every option in one forward pass and returns the probabilities, calibrated per question type with the temperatures in calibration.json.

For llama.cpp, Ollama and LM Studio, use ternary/Arbiter-8B-GGUF.

Quick start

pip install torch transformers accelerate safetensors huggingface_hub     # + peft bitsandbytes for the 4-bit adapter
hf download ternary/Arbiter-8B --local-dir Arbiter-8B
cd Arbiter-8B

from arbiter import Arbiter
from questions import OUTLOOK, EDT_TIGHT, FOMC_ORIGINAL

arbiter = Arbiter.from_pretrained(".")                    # full model (fp16)
# arbiter = Arbiter.from_pretrained(".", adapter=True)    # LoRA on Qwen/Qwen3-8B in 4-bit, as trained

passage = ("Revenue increased 15% to $8,449 million, reflecting higher shipments and higher "
           "revenue per ton. Cost of sales increased 14%.")
answer = arbiter.decide(state=passage, question=OUTLOOK, focus="Constellium")
print(answer["choice"])          # e.g. "improved"
print(answer["probabilities"])   # {"improved": ..., "worsened": ..., "unchanged": ..., "unclear": ...}

answer = arbiter.decide(state="Inflation remains elevated.",      # FOMC statement wording, 2023-24
                        question=FOMC_ORIGINAL, focus="Federal Reserve (FOMC)")
print(answer["choice"])          # e.g. "hawkish"

decide() returns a typed answer: {"type": "choice", "choice": ..., "probabilities": {...}} for choice questions, {"type": "noul", "noul": P(true), ...} for yes/no questions, and {"type": "score", "score": level, "probabilities": {...}} for ordered scales. If a state is too long, it is shortened from the end (never the options), and the answer reports state_truncated: true. arbiter.fallback_counts counts every fallback taken.

Example: implied effect vs stated sentiment

The same headline, asked two ways. Under the "implied" definition Arbiter judges what the reported facts mean for the company; under the "stated" definition it only counts a direction the headline itself expresses.

from questions import SENTIMENT_IMPLIED, SENTIMENT_STATED

headline = "Apple declares quarterly dividend of $0.25 per share"     # Apple, 2 May 2024
arbiter.decide(state=headline, question=SENTIMENT_IMPLIED, focus="Apple")
# choice: positive   {"positive": 0.897, "neutral": 0.076, "negative": 0.028}
arbiter.decide(state=headline, question=SENTIMENT_STATED, focus="Apple")
# choice: neutral    {"positive": 0.010, "neutral": 0.985, "negative": 0.005}

Example: corporate event classification

EDT_TIGHT classifies the event a company announces into twelve types. Its definitions separate genuine contracts from look-alikes such as partnerships.

from questions import EDT_TIGHT

contract = ("Palantir Technologies has been awarded a five-year, $480 million contract by the U.S. Army "
            "to expand its Maven Smart System prototype.")                    # Palantir, 29 May 2024
arbiter.decide(state=contract, question=EDT_TIGHT)
# choice: new_contract   P = 0.977

partnership = ("Apple today announced a partnership with OpenAI to integrate ChatGPT into iOS 18, "
               "iPadOS 18 and macOS Sequoia.")                                # Apple WWDC, 10 June 2024
arbiter.decide(state=partnership, question=EDT_TIGHT)
# choice: none           P = 0.975   (a partnership is not a new contract)

Outputs above are from the full model in this repository.

Requirements

• Python 3.10 or later, with torch, transformers, accelerate, safetensors and huggingface_hub.
• Full model (default): about 17 GB of GPU memory in fp16 or bf16, for example one 24 GB GPU. device_map="auto" also spreads the model over several smaller GPUs. If part of it is offloaded to CPU memory, it still runs, only more slowly.
• 4-bit adapter (adapter=True): about 6.5 GB of GPU memory, so an 8 GB GPU is enough. Also needs peft and bitsandbytes (CUDA).
• Compute: one forward pass per decision, no text generation.

Load the model through arbiter.py as shown above. The backbone also loads with plain AutoModelForCausalLM.from_pretrained, but typed decisions need the decision head, which arbiter.py applies. The Hub's automatic code snippet labels this model text-classification; it doesn't apply.

You can pass your own question in the same shape ({"type", "instructions", "criteria"}, at most 16 options). The base model's general decision ability carries over, but quality on questions outside this list has not been measured.

Evaluation

Public benchmarks

	SEntFiN (14,329)	EDT (1,933)	FOMC (496)
Arbiter v0.5, full model (this repository)	76.4%	94.2%	67.5%
Arbiter v0.5, F16 GGUF	76.8%	93.7%	66.3%
Arbiter v0.5, Q4_K_M GGUF (imatrix)	77.4%	93.6%	67.3%
Arbiter v0.4, full model	76.0%	83.5%	60.1%
Qwen3-8B, F16 GGUF	75.0%	89.3%	55.0%
Majority class	38.1%	77.2%	49.8%

95% confidence half-widths: SEntFiN ±0.7, EDT ±1.1, FOMC ±4.2 points. Original definitions, options in the published order. EDT and FOMC are scored on their held-out test splits. SEntFiN was never trained on. Paired McNemar tests against base Qwen3-8B on the same items: FOMC +11.3 points (z 4.9), EDT +4.4 (z 9.1), SEntFiN +1.9 (z 5.9), all p < 0.001.

License

Apache-2.0