|
Download README.md from OVRLab/granite-jev-1b-sampling: direct link, hf CLI and curl.
- Browser
- Download file 9.76 kB
-
https://huggingface.co/OVRLab/granite-jev-1b-sampling/resolve/main/README.md
- Command line
-
hf download hf://OVRLab/granite-jev-1b-sampling/README.md
-
curl -L -o README.md https://huggingface.co/OVRLab/granite-jev-1b-sampling/resolve/main/README.md
9.76 kB
| license: mit | |
| base_model: ibm-granite/granite-4.0-1b | |
| library_name: pytorch | |
| tags: | |
| - granite | |
| - selective-verification | |
| - inference-time-compute | |
| - research | |
| - custom-code | |
| # Selective Answer Replacement with Jev — Granite 1B runner | |
| A runnable **inference procedure**, using unchanged IBM Granite 4.0–1B weights | |
| and TypeSafe's hosted Jev verifier. The paper calls this policy **Selective Answer | |
| Replacement (SAR)** and its Jev implementation **SAR–Jev**. This release implements | |
| the original greedy-plus-one-sample policy in | |
| [Selective Answer Replacement in Frozen Language Models](https://github.com/OVRLab/Granite-Jev---Improving-Small-Language-Models-with-Selective-Verification). | |
| [Research code](https://github.com/OVRLab/jev-guided-decoding). | |
| There are no newly trained weights, adapters, internal routing changes or Jev | |
| weights in this package. It is a custom serial Python runner, not an | |
| `AutoModel` checkpoint, hosted Hugging Face chat widget or vLLM plugin. | |
| The earlier [trained-proposal artifact](https://huggingface.co/OVRLab/granite-jev-1b-research) | |
| is a different preserved experiment. | |
| ## What runs | |
| 1. Frozen Granite generates one greedy answer. | |
| 2. The same weights generate one independent sample (temperature 0.7, top-p 0.95, | |
| top-k 50), with a separate cache and a reproducible input-specific seed. | |
| 3. Exactly equal texts keep the native answer without calling Jev. Otherwise Jev | |
| 1.13.0 scores the anonymous pair against the visible task, without references. | |
| 4. Keep the sampled answer only if its score exceeds native by **strictly more | |
| than 0.05**. Return exactly that branch's generated token IDs. | |
| Jev runs during inference and sees the prompt and both candidate texts. Both | |
| Granite generations are paid for even when verification is skipped. Jev does | |
| not generate the answer tokens. Its scores are fallible and are not established | |
| as calibrated correctness probabilities. | |
| ## Download and run | |
| Use Python 3.12 (validated environment); pinned PyTorch 2.8.0 and Transformers | |
| 4.57.1. First use the | |
| [version-pinned download instructions](https://huggingface.co/OVRLab/granite-jev-1b-sampling/blob/main/INSTALL.md), | |
| which give the exact verified Hugging Face commit. Then: | |
| ```bash | |
| cd granite-jev-sampling | |
| python -m venv .venv | |
| . .venv/bin/activate | |
| python -m pip install -r requirements.txt | |
| python run_sampling.py --verify | |
| python run_sampling.py --input example.json --mode native --device cpu --max-new-tokens 128 | |
| ``` | |
| The base weights/tokenizer are downloaded directly from IBM at revision | |
| `6a7381ba1f54d684ff508d991aeb7dc580157103`. CPU execution is supported but slower; | |
| use `--device cuda` on a suitable GPU. The runner uses FP32. It does not silently | |
| quantize, truncate the prompt or change precision to fit a device. | |
| For Jev selection, put **your own** raw TypeSafe key in a private file and run: | |
| ```bash | |
| python run_sampling.py --input example.json --device cuda \ | |
| --key-file ~/.typesafe.ai/jev --output answer.json | |
| ``` | |
| Alternatively supply `TYPESAFE_API_KEY` through your runtime's secret mechanism. | |
| An explicit key file takes precedence; the default file is `~/.typesafe.ai/jev`. | |
| Native mode and artifact verification need no key. No key is stored in weights, | |
| configuration, prompts or the published artifact. Hosted use incurs the | |
| provider's charges and requires access to the pinned Jev model. | |
| Guided CLI runs require a new `--output` path, reserved before model loading or | |
| provider dispatch. Durable records retain candidate tokens, input/payload hashes, | |
| known usage and explicit failures. Inspect an interrupted record before retrying: | |
| unknown provider usage is not free, and a saved answer must not be generated again | |
| merely because its final write failed. See [runtime and security guidance](SECURITY.md). | |
| ## Input and limits | |
| ```json | |
| {"id":"my-question-1","task":"math","prompt":"What is 9 + 6? Give just the number."} | |
| ``` | |
| Allowed task rubrics: `instruction`, `math`, `reasoning`, `code`, `code_output`, | |
| `safety`, `grounding`. These names choose verification instructions; they are | |
| not assertions that a question belongs to an independently validated benchmark. | |
| Include all relevant visible source material in `prompt`. Extra fields such as | |
| reference answers are rejected. An omitted ID is derived from the input. | |
| Saved results include the task rubric, stable ID and canonical input hash; guided | |
| judgments also record a hash of the exact verifier payload. Keep the input and | |
| pinned release revision with the result so the judgment contract can be audited. | |
| `--replicate` chooses 3701, 3702 or 3703; each is a separate sample-and-verification | |
| draw, not best-of-three. Outputs are capped at 2,048 tokens per branch and total | |
| context at 32,768 tokens. The default output cap is 1,024. `--max-seconds` is a | |
| bounded request deadline. One runner owns one serial request; no generated code | |
| is executed. A user prompt is sent to Jev only on disagreement. | |
| The API is synchronous: invoke it outside a running asyncio event loop. An | |
| event-loop rejection happens before provider dispatch and records zero API attempts. | |
| ## Results in the November 2026 paper | |
| The prospective study compares greedy generation with one independent sample | |
| selected by Jev. Accuracy (%) uses the registered strict final-choice parser; | |
| absolute gains are percentage points, with paired 95% confidence intervals. | |
| | Benchmark | Granite backbone | Greedy | SAR–Jev | Absolute gain [95% CI] | | |
| | --- | --- | ---: | ---: | ---: | | |
| | ARC-Challenge | 4.0–1B | 66.23 | 71.25 | +5.02 [4.02, 6.05] | | |
| | ARC-Challenge | 4.2–3B standard | 83.91 | 86.81 | +2.90 [2.12, 3.74] | | |
| | CommonsenseQA | 4.0–1B | 48.89 | 52.99 | +4.10 [3.22, 5.02] | | |
| | CommonsenseQA | 4.2–3B standard | 73.05 | 75.10 | +2.05 [1.28, 2.84] | | |
| There are 1,069 ARC-Challenge and 1,221 CommonsenseQA questions per backbone, | |
| each with three paired draws sharing the greedy answer; these are not three | |
| independent datasets. All four comparisons have Holm-adjusted p < .001. | |
| The 3B rows describe separate paper experiments; **this download runs 1B only**. | |
| The paper also evaluates Qwen and Phi as generators and a common prospective | |
| GSM8K protocol: numeric accuracy rises 70.89→77.79% for Granite, 89.92→92.22% | |
| for Phi and 91.81→93.56% for Qwen under the fixed 0.05 rule. These studies use | |
| separate pinned research runners and grading contracts, not additional backbones | |
| supported by this download. Development-only margin selection did not establish | |
| a better held-out correction–harm loss. A subsequent saved-candidate experiment | |
| favors two samples on Granite/Qwen numeric accuracy but loses Phi joint | |
| correctness/formatting. It does not change this release's greedy-plus-sample default. | |
| A local open-verifier replay on those same mathematical pairs leaves all six | |
| construction/interaction tests inconclusive and retains a larger Phi joint-score | |
| loss. It is separate from the earlier ARC/CSQA Open control. The local verifier | |
| is research code, not an additional runtime bundled in this download. | |
| The paper also reports exploratory reasoning, instruction-following, coding, | |
| knowledge, multilingual, tool-use and safety evaluations. See [RESULTS.md](RESULTS.md) | |
| for their scope, controls and links to numerical exports. Improvements are not | |
| uniform across controls: three-sample voting (Vote-3) scores 62.90% on | |
| 1B CommonsenseQA under strict grading, above SAR–Jev's 52.99%. | |
| In the primary records Jev is called on 97.55% of pairs and replaces the greedy | |
| answer on 13.85%. “Selective” describes answer replacement, not infrequent API | |
| use. Separate serial measurements on 100 previously evaluated questions show | |
| roughly threefold median latency versus greedy; hosted Jev computation adds | |
| cost. The evidence does not establish a matched-total-cost advantage. | |
| ## Reproducibility and release validation | |
| This documentation revision updates the generic method name and completed | |
| evidence summaries. It retains the prior maintenance dependency minimums and | |
| security guidance. The Python runtime, selection rule, source provenance and | |
| IBM weight revision are unchanged. [VALIDATION.md](VALIDATION.md) separates | |
| current offline maintenance checks from the preserved October real-model and | |
| live-provider checks; those live checks were not repeated for this update. | |
| This prompt runner is not the full benchmark harness: task templates, parsers, | |
| stopping limits and agreement rules are cohort-specific. Use the paper's protocols | |
| and pinned source for numerical reproduction. `source_manifest.json` records | |
| frozen generation-function provenance; `artifact_manifest.json` and `SHA256SUMS` | |
| check file integrity, not publisher authenticity. Pin the full Hugging Face commit. | |
| Scores can vary across devices and provider executions. | |
| The demo makes at most **one provider attempt** per request. Failed or invalid | |
| judgments remain errors, not successful native fallbacks. Benchmark campaigns | |
| have separate bounded five-minute outage recovery; this runner does not. | |
| ## License and limitations | |
| OVRLab code is MIT; IBM base weights are separately Apache 2.0. Jev is a hosted | |
| service with separate terms, availability and charges. No Jev weights or | |
| benchmark text are redistributed. See `NOTICE.md` and `SECURITY.md`. | |
| This system can produce incorrect answers and harmful corrections. It adds | |
| generation and API overhead. Current evidence does not establish universal | |
| improvement, superiority at equal resources, a new internal LLM architecture, | |
| or replacement of a larger model. No endorsement by IBM or TypeSafe is implied. | |
| Linux and macOS are supported; native Windows is outside the runtime boundary | |
| (use a Linux environment such as WSL2). Read [SECURITY.md](SECURITY.md) before | |
| networked use, including the known advisories in the pinned inference libraries. | |