OVRLab/granite-jev-1b-sampling

Selective Answer Replacement with Jev — Granite 1B runner

A runnable inference procedure, using unchanged IBM Granite 4.0–1B weights and TypeSafe's hosted Jev verifier. The paper calls this policy Selective Answer Replacement (SAR) and its Jev implementation SAR–Jev. This release implements the original greedy-plus-one-sample policy in Selective Answer Replacement in Frozen Language Models. Research code.

There are no newly trained weights, adapters, internal routing changes or Jev weights in this package. It is a custom serial Python runner, not an AutoModel checkpoint, hosted Hugging Face chat widget or vLLM plugin. The earlier trained-proposal artifact is a different preserved experiment.

What runs

1. Frozen Granite generates one greedy answer.
2. The same weights generate one independent sample (temperature 0.7, top-p 0.95, top-k 50), with a separate cache and a reproducible input-specific seed.
3. Exactly equal texts keep the native answer without calling Jev. Otherwise Jev 1.13.0 scores the anonymous pair against the visible task, without references.
4. Keep the sampled answer only if its score exceeds native by strictly more than 0.05. Return exactly that branch's generated token IDs.

Jev runs during inference and sees the prompt and both candidate texts. Both Granite generations are paid for even when verification is skipped. Jev does not generate the answer tokens. Its scores are fallible and are not established as calibrated correctness probabilities.

Download and run

Use Python 3.12 (validated environment); pinned PyTorch 2.8.0 and Transformers 4.57.1. First use the version-pinned download instructions, which give the exact verified Hugging Face commit. Then:

cd granite-jev-sampling
python -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
python run_sampling.py --verify
python run_sampling.py --input example.json --mode native --device cpu --max-new-tokens 128

The base weights/tokenizer are downloaded directly from IBM at revision 6a7381ba1f54d684ff508d991aeb7dc580157103. CPU execution is supported but slower; use --device cuda on a suitable GPU. The runner uses FP32. It does not silently quantize, truncate the prompt or change precision to fit a device.

For Jev selection, put your own raw TypeSafe key in a private file and run:

python run_sampling.py --input example.json --device cuda \
  --key-file ~/.typesafe.ai/jev --output answer.json

Alternatively supply TYPESAFE_API_KEY through your runtime's secret mechanism. An explicit key file takes precedence; the default file is ~/.typesafe.ai/jev. Native mode and artifact verification need no key. No key is stored in weights, configuration, prompts or the published artifact. Hosted use incurs the provider's charges and requires access to the pinned Jev model.

Guided CLI runs require a new --output path, reserved before model loading or provider dispatch. Durable records retain candidate tokens, input/payload hashes, known usage and explicit failures. Inspect an interrupted record before retrying: unknown provider usage is not free, and a saved answer must not be generated again merely because its final write failed. See runtime and security guidance.

Input and limits

{"id":"my-question-1","task":"math","prompt":"What is 9 + 6? Give just the number."}

Allowed task rubrics: instruction, math, reasoning, code, code_output, safety, grounding. These names choose verification instructions; they are not assertions that a question belongs to an independently validated benchmark. Include all relevant visible source material in prompt. Extra fields such as reference answers are rejected. An omitted ID is derived from the input. Saved results include the task rubric, stable ID and canonical input hash; guided judgments also record a hash of the exact verifier payload. Keep the input and pinned release revision with the result so the judgment contract can be audited.

--replicate chooses 3701, 3702 or 3703; each is a separate sample-and-verification draw, not best-of-three. Outputs are capped at 2,048 tokens per branch and total context at 32,768 tokens. The default output cap is 1,024. --max-seconds is a bounded request deadline. One runner owns one serial request; no generated code is executed. A user prompt is sent to Jev only on disagreement. The API is synchronous: invoke it outside a running asyncio event loop. An event-loop rejection happens before provider dispatch and records zero API attempts.

Results in the November 2026 paper

The prospective study compares greedy generation with one independent sample selected by Jev. Accuracy (%) uses the registered strict final-choice parser; absolute gains are percentage points, with paired 95% confidence intervals.

Benchmark	Granite backbone	Greedy	SAR–Jev	Absolute gain [95% CI]
ARC-Challenge	4.0–1B	66.23	71.25	+5.02 [4.02, 6.05]
ARC-Challenge	4.2–3B standard	83.91	86.81	+2.90 [2.12, 3.74]
CommonsenseQA	4.0–1B	48.89	52.99	+4.10 [3.22, 5.02]
CommonsenseQA	4.2–3B standard	73.05	75.10	+2.05 [1.28, 2.84]

There are 1,069 ARC-Challenge and 1,221 CommonsenseQA questions per backbone, each with three paired draws sharing the greedy answer; these are not three independent datasets. All four comparisons have Holm-adjusted p < .001. The 3B rows describe separate paper experiments; this download runs 1B only.

The paper also evaluates Qwen and Phi as generators and a common prospective GSM8K protocol: numeric accuracy rises 70.89→77.79% for Granite, 89.92→92.22% for Phi and 91.81→93.56% for Qwen under the fixed 0.05 rule. These studies use separate pinned research runners and grading contracts, not additional backbones supported by this download. Development-only margin selection did not establish a better held-out correction–harm loss. A subsequent saved-candidate experiment favors two samples on Granite/Qwen numeric accuracy but loses Phi joint correctness/formatting. It does not change this release's greedy-plus-sample default. A local open-verifier replay on those same mathematical pairs leaves all six construction/interaction tests inconclusive and retains a larger Phi joint-score loss. It is separate from the earlier ARC/CSQA Open control. The local verifier is research code, not an additional runtime bundled in this download.

The paper also reports exploratory reasoning, instruction-following, coding, knowledge, multilingual, tool-use and safety evaluations. See RESULTS.md for their scope, controls and links to numerical exports. Improvements are not uniform across controls: three-sample voting (Vote-3) scores 62.90% on 1B CommonsenseQA under strict grading, above SAR–Jev's 52.99%.

In the primary records Jev is called on 97.55% of pairs and replaces the greedy answer on 13.85%. “Selective” describes answer replacement, not infrequent API use. Separate serial measurements on 100 previously evaluated questions show roughly threefold median latency versus greedy; hosted Jev computation adds cost. The evidence does not establish a matched-total-cost advantage.

Reproducibility and release validation

This documentation revision updates the generic method name and completed evidence summaries. It retains the prior maintenance dependency minimums and security guidance. The Python runtime, selection rule, source provenance and IBM weight revision are unchanged. VALIDATION.md separates current offline maintenance checks from the preserved October real-model and live-provider checks; those live checks were not repeated for this update.

This prompt runner is not the full benchmark harness: task templates, parsers, stopping limits and agreement rules are cohort-specific. Use the paper's protocols and pinned source for numerical reproduction. source_manifest.json records frozen generation-function provenance; artifact_manifest.json and SHA256SUMS check file integrity, not publisher authenticity. Pin the full Hugging Face commit. Scores can vary across devices and provider executions.

The demo makes at most one provider attempt per request. Failed or invalid judgments remain errors, not successful native fallbacks. Benchmark campaigns have separate bounded five-minute outage recovery; this runner does not.

License and limitations

OVRLab code is MIT; IBM base weights are separately Apache 2.0. Jev is a hosted service with separate terms, availability and charges. No Jev weights or benchmark text are redistributed. See NOTICE.md and SECURITY.md.

This system can produce incorrect answers and harmful corrections. It adds generation and API overhead. Current evidence does not establish universal improvement, superiority at equal resources, a new internal LLM architecture, or replacement of a larger model. No endorsement by IBM or TypeSafe is implied.

Linux and macOS are supported; native Windows is outside the runtime boundary (use a Linux environment such as WSL2). Read SECURITY.md before networked use, including the known advisories in the pinned inference libraries.