Text Classification
GGUF
ollaya
ollama
trainaya
decision-model
system-1
llm-logits-v1
calibrated
Mixture of Experts
imatrix
conversational
push-26b / README.md
solarpush's picture
Upload folder using huggingface_hub
c68be8d verified
|
Raw History Blame Contribute Delete
12.7 kB
metadata
license: apache-2.0
library_name: gguf
pipeline_tag: text-classification
base_model: google/gemma-4-26B-A4B-it
base-model: google/gemma-4-26B-A4B-it
language:
  - en
  - de
  - fr
  - es
  - multilingual
tags:
  - ollaya
  - ollama
  - trainaya
  - decision-model
  - system-1
  - llm-logits-v1
  - gguf
  - calibrated
  - moe
datasets:
  - ZefanCai/Open-Jev-v1.1
  - SargeDev/jev-distill-corpus-v3
  - tasksource/tasksource-jev-typed-decisions

push:26b — Calibrated System-1 Decision Model (26B-A4B MoE GGUF)

push:26b (solarpush/push-26b) is a high-precision, calibrated System-1 Decision Model (llm-logits-v1) designed for single-forward-pass structured decision-making across three universal primitives:

  • choice — Multi-way categorical selection (A..Z, 2 to 26 options),
  • noul — Binary epistemic / policy / safety / entailment verification (Yes vs No),
  • score — Ordinal rubric & severity evaluation (0..9, 3- to 5-level calibrated scales).

Built on a sparse 26B total / 4B active parameter Mixture-of-Experts (MoE) GGUF backbone (UD-Q3_K_M, 12.42 GiB on disk, ~15.0 GB VRAM) with an untied, subspace-orthogonal Q6_K (ttype=14) decision head and cardinality-conditioned temperature scaling (temperature_by_options), push:26b delivers frontier-grade accuracy and calibration in ~223 ms p50 on a single 16 GB consumer GPU.


1. Official Ollaya Decision Benchmark (bench.ollaya.dev — 14 Tasks, 5,880 Questions)

Evaluated strictly zero-shot / out-of-domain across all 14 official bench.ollaya.dev tasks (3,880 Public benchmark questions across 13 datasets + 2,000 typed-decisions workflow questions across 400 multi-question JSON states = 5,880 total questions, zero benchmark train/dev/test overlap):

Model Overall Skill [95% CI] Public Skill (13 Tasks) Public Macro Acc Typed-Decisions Acc (Skill) Public Pooled ECE (lower is better) Typed ECE (lower is better) Global Brier Latency (p50) VRAM License
push:26b (26B-A4B MoE) 62.7 [60.8, 64.7] 63.1 75.4% 71.4% (58.1) 2.8% (0.028) 2.0% (0.020) 0.368 223 ms 15.0 GB Apache-2.0
winnow:e4b (Reference 4B) 59.8 [58.0, 61.5] 58.9 72.7% 72.4% (59.5) 3.4% (0.034) 3.2% (0.032) 0.380 46 ms 6.5 GB Apache-2.0
push:3b (Adapted 3B) 46.0 [43.9, 48.2] 47.1 64.8% 53.9% (32.5) 3.1% (0.031) 15.3% (0.153) 0.520 86 ms 4.2 GB Apache-2.0

Accuracy by Decision Primitive (5,880 Questions)

  • Choice (Multi-Option Selection): 76.8%
  • Noul (Binary Verification & Guardrails): 82.2%
  • Score (Ordinal Rubric Evaluation): 60.5%

2. Per-Task Breakdown (14 Tasks — 5,880 Questions)

Benchmark Task Primitive Questions push:26b Accuracy push:26b Skill (0–100) winnow:e4b Accuracy (Skill) Delta vs winnow:e4b push:26b ECE (10-bin) push:26b Brier Median Latency
squad2 (Unanswerable QA) noul 299 87.3% (up to 87.6%) 74.6 79.6% (59.2) +7.7% (+15.4 Skill) 0.064 0.211 122 ms
massive-en-US (18-Way Intent) choice 350 86.9% 86.1 84.6% (83.7) +2.3% (+2.4 Skill) 0.047 0.222 124 ms
summeval-consistency (Factual Consistency) score 144 86.1% 82.6 83.3% (79.2) +2.8% (+3.4 Skill) 0.187 0.274 159 ms
massive-de-DE (18-Way Intent DE) choice 350 85.4% 84.6 83.7% (82.8) +1.7% (+1.8 Skill) 0.071 0.246 125 ms
boolq (Reading Verification) noul 300 84.0% (up to 85.3%) 68.0 86.0% (72.0) -2.0% 0.096 0.228 107 ms
civil_comments (Toxicity Detection) noul 300 82.0% (up to 82.3%) 64.0 75.7% (51.3) +6.3% (+12.7 Skill) 0.119 0.268 95 ms
paws (Adversarial Paraphrase) noul 250 80.4% 60.8 84.0% (68.0) -3.6% 0.058 0.300 99 ms
multinli (3-Way NLI) choice 299 79.6% (up to 81.3%) 69.4 82.9% (74.4) -3.3% 0.044 0.320 92 ms
aegis2 (AI Safety Guardrail) noul 250 78.0% (up to 79.2%) 56.0 80.8% (61.6) -2.8% 0.094 0.324 101 ms
pubmedqa (Biomedical 3-Way QA) choice 250 74.0% (up to 74.4%) 61.0 69.6% (54.4) +4.4% (+6.6 Skill) 0.076 0.359 140 ms
vitaminc-dev (3-Way Fact Verification) choice 599 72.5% 58.7 69.3% (53.9) +3.2% (+4.8 Skill) 0.062 0.414 102 ms
typed-decisions (400 JSON Workflows) mixed 2000 71.4% (up to 71.8%) 58.1 72.4% (59.5) -1.0% 0.020 0.402 294 ms
summeval-relevance (5-Level Relevance) score 240 48.3% 35.4 44.6% (30.7) +3.7% (+4.7 Skill) 0.046 0.659 189 ms
helpsteer2 (5-Level Helpfulness) score 249 40.2% (up to 41.4%) 25.2 37.3% (21.7) +2.9% (+3.5 Skill) 0.154 0.741 167 ms
OVERALL (14 Tasks) All 5880 75.1% (75.4% Public) 62.7 72.6% (59.8) +2.7% Public (+2.9 Skill) 0.083 (0.028 Pooled) 0.368 223 ms

typed-decisions Enterprise Workflow Breakdown (2,000 Questions / 400 JSON States)

  • customer_service: 77.6%
  • invoice_processing: 74.6% (up to 76.0%)
  • security_incidents: 72.6%
  • agent_trace_observability: 61.6% (up to 64.2%)
  • Calibration: ECE = 0.020 (2.0%) | Soft-CE = 1.010

3. Training & Calibration Methodology (100% Zero-Benchmark-Leakage)

push:26b was adapted with Trainaya using strictly generalist, out-of-benchmark decision datasets (ZefanCai/Open-Jev-v1.1, SargeDev/jev-distill-corpus-v3, tasksource/tasksource-jev-typed-decisions with all 14 benchmark dataset sources strictly excluded, plus balanced cyclic $3 \times 3$ ABC counterfactual triplets):

  1. Untied Q6_K (ttype=14) Output Head Materialization:
    • Unties output.weight ([262144, 2816]) from token_embd.weight so the 62 decision rows (A..Z, space-prefixed ▁A..▁Z, and 0..9) are updated in high-precision Q6_K without perturbing backbone token embeddings.
  2. Biorthogonal Dual Subspace Projection (U, U_s, vt_bin, vt_score_prot):
    • Solves separate orthogonal projections for 3-way NLI, 3-way Fact Verification, 3-way QA, 4- and 5-option workflow choices, binary noul verification, and 5-level ordinal rubrics so updates to multi-option choice produce zero drift (0.000000) on protected binary noul and ordinal score subspaces.
  3. Fisher-ARD Kernel Ridge + Composite Soft-Target Distillation (KL + Brier + RPS + Ordinal + Softcap):
    • Blends hard gold labels with soft teacher distributions (0.65 * y_hard + 0.35 * y_soft) across multi-scale ridge ensembles (λ ∈ {20, 40, 75}) followed by subspace-protected composite KL + Brier + Ranked Probability Score (RPS) polish respecting the architecture's native final_logit_softcap = 30.0.
  4. Cardinality-Conditioned Temperature Calibration (temperature_by_options):
    • Calibrates distinct post-softcap temperatures per primitive and option count (noul:2 = 8.91, choice:2 = 6.35, choice:3-5 = 8.2485, choice:6-10 = 6.0325, choice:11+ = 4.6355, score:3-5 = 10.3), achieving a Pooled Public ECE of 2.8% (0.028).

4. Repository Contents

File Size Description
model.gguf 12.42 GiB 26B-A4B MoE GGUF (UD-Q3_K_M backbone + untied Q6_K calibrated output.weight decision head)
decision.json 2.5 KB Ollaya llm-logits-v1 decision specification (labels, template, temperature_by_options, llama runtime config)
calibration.json 751 B Post-hoc temperature scaling parameters and out-of-fold / post-quantization accuracy metrics
ollaya_config.json 302 B Ollaya runtime manifest configuration
Modelfile 559 B Deterministic single-token Ollama Modelfile (temperature 0, top_k 1, num_predict 1)

5. Quickstart (Ollaya & Ollama)

Option A: Register in both Ollaya and Ollama via trainaya

git clone https://huggingface.co/solarpush/push-26b
cd push-26b
trainaya register --model . --name push:26b --runtime both

Option B: Query via Ollaya /v1/systemone API

curl -s http://127.0.0.1:11435/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "model": "push:26b",
    "state": "Context: Aspirin inhibits platelet aggregation via irreversible COX-1 acetylation. In a randomized trial of 1,200 patients, low-dose aspirin reduced recurrent thrombotic events by 24% (p = 0.003).",
    "questions": [
      {
        "id": "q1",
        "type": "noul",
        "question": "Is the reduction in recurrent thrombotic events statistically significant at p < 0.01?",
        "options": {"true": "Yes, statistically significant", "false": "No, not significant"}
      }
    ]
  }'

Option C: Run with Ollama

ollama create push:26b -f Modelfile
ollama run push:26b