FullyAutomated/granite-system1-350m-experimental

Granite System1 350M — experimental decision scorer

A non-generative, variable-option decision scorer adapted from IBM Granite 4.0 350M. Given a context, question and offered options, it returns a score distribution, an option ID or semantic NONE. It does not generate an answer or execute an action.

Experimental result, not a qualified general decision model. On 768 locally fresh evaluation cases, the larger canonical training course improved accuracy from 315/768 (41.0%) to 429/768 (55.9%). The model nevertheless failed five of eight predefined acceptance checks. Public NLI accuracy remained 39.3%; the synthetic budget family scored 7/32. Do not use it as a permission, payment, safety or autonomous execution authority.

Professional report source: https://github.com/JH427/fullyautomated.enterprises/blob/9ad2d6843296814f6422027da3290dc1007c07dc/src/content/labs/granite-system1-350m.md

The website edition is prepared but not deployed. The report source and this model card are the currently published research account; website publication is deferred.

What this repository contains

• The exact trained adapter/head tensors used in the reported comparison, exported with a portable inference configuration and source-integrity checks.
• Custom PyTorch inference code, an example request and runtime tests.
• Exact training, development and consumed confirmation data, under separate dataset licensing terms.
• Original raw control/trained predictions and a small standard-library auditor that recomputes decision counts from those observations.
• Machine-readable results, the training/evaluation protocol and explicit limitations.

This is not a complete base-model checkpoint, a chat model, a standard PEFT adapter, or a Transformers AutoModel.from_pretrained() release. Use the supplied loader. The IBM backbone/tokenizer are fetched separately at a pinned revision, or supplied as a verified local directory. No server or agent harness is installed.

Quick start

Download this repository into its own directory using the Hugging Face CLI or the website:

hf download FullyAutomated/granite-system1-350m-experimental --local-dir granite-system1-350m
cd granite-system1-350m
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python system1.py --model-dir . --input examples/request.json

The first inference command needs access to the pinned public IBM model if it is not already cached. To reuse an existing matching backbone without a network fetch:

python system1.py --model-dir . --base-dir ./base-model --input examples/request.json

./base-model must contain the original pinned model and tokenizer assets; it is not the adapter directory. The loader verifies the model and tokenizer hashes. Read the custom source before executing it. No trust_remote_code=True or model-provided Python code is required.

Decision contract

Input is a JSON object containing context, question, and 2–16 options, each with an opaque string id and a natural-language description. IDs must be unique; duplicate descriptions and over-limit inputs are rejected. The interface accepts up to the retained native 32,768-token capacity, but long-context usefulness and a complete full-window execution were not demonstrated. Training inputs were limited to 768 tokens; confirmation inputs were at most 246 tokens.

Output includes all offered option IDs plus JSON null for NONE, logits, raw softmax probabilities and the chosen ID. Scores are returned in caller order; encoding and tie-breaking use exact-description canonical order with NONE last.

NONE means that no offered option satisfies the criteria. It does not mean uncertainty, low confidence, refusal, or permission to act. Raw softmax probabilities are uncalibrated, and uncertainty is unavailable. A confident-looking wrong answer remains a wrong answer.

Architecture and provenance

• Base: ibm-granite/granite-4.0-350m, revision bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2, Apache-2.0.
• Base safetensors SHA-256: 21bc1ccaf0eb6a8a503addd93cb7422df71129de0a641a5525e4a665813b6dfc.
• 352,379,904 frozen backbone parameters; 28 attention layers and no experts in this checkpoint, despite the Transformers hybrid class name.
• Rank-8, alpha-16 q/v LoRA modules in the last four attention blocks; shared 1024→128→1 GELU option head and a separate NONE head. Total trainable parameters: 369,154.
• Candidate scores use option-segment last-token readouts; NONE uses the final-segment last-token state. This is a causal sequence model: canonical ordering removes opaque-ID/order perturbations, not candidate-prefix dependence.
• Exact option descriptions are sorted before encoding. Opaque IDs are mapped outside the model. Identical descriptions are invalid input rather than a no-match case.
• Adapter/head safetensors SHA-256: adffd25386d5615acac4c2158789ff4af93d21fa4f73a33f66629d4673911b82.

The public export changes packaging, not the trained tensors or canonical decision representation. It is separate from the original local format-2 serializer. There is no claim of architectural novelty.

Release verification: the package was copied outside the research directory and exercised in a fresh CPU process with Hub/Transformers offline mode enabled and existing pinned base assets. Eight saved cases matched the original CPU predictions exactly, including complete score distributions; all eight also retained exact mapped distributions under option reversal and ID renaming. The standalone example CLI ran successfully. Fifteen release tests passed (11 inference tests and four result-auditor tests, plus 17 inference subtests). This was not a fresh dependency installation. See release-evidence/portability.json.

Training procedure

The reported course warm-started from the preceding canonical learnability adapter, not the unadapted base. That small diagnostic used 48 unique training examples and 48 development examples, with 24 training repetitions: final unique training accuracy was 44/48 and development accuracy 21/48. These are learnability diagnostics, not general-quality estimates.

The larger course used 16,384 training row IDs (15,904 distinct visible states) and 384 development rows (383 distinct states). It completed 2,048 updates, effective batch size eight, with each training row ID exposed once. Each balanced batch included two semantic NONE cases. Cross-entropy training used AdamW, adapter LR 1e-5, head LR 5e-5, weight decay 0.01 and gradient clipping 1.0. The backbone remained frozen.

Full development NLL, including step zero, selected the checkpoint. Step 2,048 was selected. Development accuracy changed from 179/384 to 215/384; NLL changed from approximately 1.3305 to 0.9614. Reported training metrics used a 128-row sample, not the full training set.

The course ran in float32 on an AMD Radeon 8060S. Training helper time was 1,000.411 seconds; whole training-job time was 1,007.386 seconds. The paired fresh evaluation took 84.683 whole-job seconds. Combined job wall time was 1,092.069 seconds, excluding preparation, implementation and audits. These are not inference throughput benchmarks or full-model pretraining costs.

Evaluation

The confirmation set contained 384 synthetic structured decisions across 12 trained logical families and 384 adapted public SNLI cases. There were 192 NONE targets. Membership was frozen and checked against previously used local data before comparison. It is now consumed evaluation data and must not be called fresh for future iterations.

Measure	Canonical learnability control	Larger canonical course
Overall accuracy	315/768 — 41.0%	429/768 — 55.9%
Structured decisions	209/384 — 54.4%	278/384 — 72.4%
Public NLI	106/384 — 27.6%	151/384 — 39.3%
Supported-option accuracy	33.7%	52.4%
NONE precision	37.0%	57.0%
NONE recall	63.0%	66.1%
Mean NLL	1.3676	0.9448

An always-NONE rule scores 25.0% on this deliberately constructed mixture. Always selecting the first offered option scores approximately 27.2%; a uniform random choice over offered options plus NONE has expected accuracy approximately 26.5%. These are weak reference baselines, not a competitive symbolic decision system. The 41.0% control is an earlier trained checkpoint, not an unadapted-base baseline.

Only three of eight prospective checks passed: strict improvement over control and exact full-distribution agreement for 32 option-reversal and 32 opaque-ID-renaming probes, for both checkpoints. Gates requiring at least 75% overall accuracy, supported accuracy, NONE precision and NONE recall, plus at least 65% accuracy in each domain, failed. Exact tested invariance is not evidence of decision correctness or universal perturbation robustness.

Inspect release-metrics.json for all family/domain metrics and release-evidence/ for original per-case observations.

Inspect and reproduce

Audit the recorded result without loading a model:

python release-evidence/audit_results.py

This verifies the frozen test-data hash, case membership and labels, checks choices against canonical argmax of the original logits, and recomputes overall/supported/NONE counts. It does not rerun inference or verify each raw softmax value. It fails if the declared decision metrics differ from the recomputed values.

For tests, install pytest separately and run:

python -m pip install pytest
python -m pytest tests release-evidence/tests -q

The repository includes the exact larger-course data and a detailed protocol, but it does not provide or claim an independently rerun end-to-end training reproduction. Some historical code, the warm-start artifact and earlier development inventories are not redistributed. Do not mistake a recipe or saved logits for a fresh training run. Earlier local work used Python 3.12 / PyTorch 2.15.0+cpu and Python 3.13 / PyTorch 2.13.0+rocm7.2, with Transformers 5.22.1, safetensors 0.8.0 and huggingface_hub 2.4.0. No broad platform compatibility or clean dependency-install result is implied by those version records.

Data provenance and licensing

Public NLI derives from SNLI, Hugging Face dataset stanfordnlp/snli at revision cdb5c3d5eed6ead6e5a341c8e56e669bb666725b. Labels are mapped to entailment, neutral and contradiction; NONE is assigned only where the gold relation is omitted from the offered options. Neutral is not redefined as NONE.

The transformed data files under release-evidence/data/ are CC-BY-SA-4.0, separately from the Apache-2.0 software and adapter/head weights. See NOTICE.md and release-evidence/data/LICENSE.md for attribution, adaptation details and share-alike terms. No photographs are included.

Bowman, Angeli, Potts and Manning (2015), A large annotated corpus for learning natural language inference. SNLI premises originate from Flickr30k captions (Young, Lai, Hodosh and Hockenmaier, 2014).

Limitations and intended use

Research, inspection and bounded offline experiments only. This is not a dependable general decision model.

• Fresh instances in trained families, not unseen-family transfer. Local split checks do not establish absence of base-model pretraining exposure.
• The source parquet lacked image IDs, so image-level independence was not verified. This transformed evaluation is not an official SNLI benchmark score.
• Dataset mixture, NONE prevalence and task templates were constructed; performance is not an estimate for arbitrary production traffic.
• Comparisons across the development history were not controlled ablations: representation, data and training changed. The model's capacity ceiling has not been established.
• No calibration, uncertainty quality, selective-accuracy or net agent-cost savings study was completed.
• Native 32,768-token capacity was preserved, but a previous full-window CPU forward timed out. Neither useful long-context decisions nor full-window execution are established.
• No quantization, production integration or deployment-readiness claim. Source/runtime integrity checks detect accidental mismatches; they are not a sandbox or proof that downloaded code is safe.

Citation

@misc{fae2026granitesystem1,
  author = {{Fully Automated Enterprises}},
  title = {Adapting Granite 4.0 350M for Non-Generative, Variable-Option Decisions},
  year = {2026},
  url = {https://huggingface.co/FullyAutomated/granite-system1-350m-experimental},
  note = {Experimental research release; predefined general-quality criteria not met}
}