--- language: - en base_model: ibm-granite/granite-embedding-reranker-english-r2 base_model_relation: finetune license: apache-2.0 library_name: onnxruntime tags: - tokenfold - context-compression - extractive-selection - experimental - custom-model - onnx - safetensors --- # Tokenfold Select v2 ### Build smaller contexts. Keep the original words. **Query-aware context selection in a compact, 98.9M-parameter model: ONNX by default, Safetensors for PyTorch workflows.** LLM workflows collect more context than every request needs. Tokenfold Select v2 helps your application choose what to keep: it scores text groups together with the query, then hands those scores to an allocator that assembles a smaller context from the original text. No generated summaries or paraphrases. **Experimental ONNX + Safetensors preview · English · Local inference** ## Why Tokenfold Select v2? - **Score groups together.** One joint encoder pass scores all groups in an admitted single-window request, rather than encoding each query/span pair separately. - **Keep the source intact where it counts.** The extractive workflow retains selected original groups; your allocator controls required content, source order and the final token budget. - **Use a compact specialist.** A trained 12-layer encoder with a shared group scoring head—not a generative LLM or the v1 LoRA adapter. - **Run locally.** Offline-capable inference with caller-provisioned weights and explicit precision settings. No hosted model service is required. - **Inspect every selection.** Ordered group scores give your application a clear interface for ranking, allocation and debugging. ```text Query + original text groups ↓ Select v2 group scores ↓ Allocator + required-content rules + token budget ↓ Selected context, in original wording ``` For developers exploring context selection in retrieval, tool/API responses, coding assistants and agent histories. The model returns **scores**, not a compressed document; grouping and budget enforcement remain in your application. ## Tokenfold Select v2: reference performance Using the PyTorch/CUDA FP16 reference runtime (not ONNX) on the JSON development fixture at 50% retention, Tokenfold Select v2 recorded **150 ms warm p50 compression latency**, versus **208 ms for Select v1**—about **28% lower**. The table shows **p50 / p95 milliseconds** at the same retention target: | Development fixture | Tokenfold Select v2 **FP16** | Tokenfold Select v1 FP32 | Headroom Kompress v2 FP32 | | --- | ---: | ---: | ---: | | JSON | **150.199 / 161.996** | 207.644 / 215.143 | Budget refusal: 0/6 completions | | Code | **268.713 / 282.441** | 451.916 / 466.936 | Budget refusal: 0/6 completions | | Logs | **1842.744 / 1862.359** | 3760.490 / 3788.029 | Budget refusal: 0/6 completions | | Multi-turn | 267.051 / 271.358 | 563.040 / 568.882 | **94.577 / 97.534** | | Retrieval | **773.447 / 803.698** | 2046.032 / 2070.185 | 2733.641 / 2751.180 | **Scope matters:** one exposed source per family, six repeated requests per arm and budget, on one local RTX 5080 using the PyTorch/CUDA reference runtime, not ONNX. These are empirical warm development results, not independent-source confidence bounds or proof of downstream answer quality. Kompress is faster on multi-turn where both complete; its budget refusals are failures, not evidence of a successful-compression speedup for Select v2. The complete comparison covered **180/180 requests** across two retention targets: 132 valid compressions and 48 healthy Kompress budget refusals, with no omissions. Refusals retain infinite completion latency in the failure-aware summaries. The clock includes fresh grouping, tokenization, scoring, native allocation/IPC and output recount; it excludes loading, warmup and file I/O. This measures the standalone Kompress compressor, not the full Headroom pipeline. [See all workload/budget results and methodology](evidence/development-compression.json). ## Run Tokenfold Select v2 with ONNX The staged model is a **FP32 ONNX graph plus its external weights**: `model.onnx` and `model.onnx.data`. Keep both files together. No pickle-based `.pt` checkpoint is included in this upload candidate. The optional Safetensors file is separate; ONNX inference does not require PyTorch. | Component | Current scope | | --- | --- | | Model | 98,864,641 parameters, 12 encoder layers, ONNX opset 18 | | Graph IO | int64 `input_ids` / `attention_mask`; FP32 `token_scores`, shape `[1, tokens]` | | Admission | One complete window; at most 8,192 encoded tokens and 512 groups | | Runtime | ONNX Runtime, explicit CPUExecutionProvider; no implicit provider fallback | | Software | Python 3.12 and the [pinned ONNX environment](environment-requirements.txt) | The ONNX graph and reference adapter passed **6/6 synthetic CPU parity checks**, including two real packaged-loader requests and dynamic sequence lengths up to 2,048 tokens / 512 pooled group ranges. Maximum absolute score error was **0.0000878**, below the predeclared 0.0001 tolerance; ranking agreement was approximately 1.0. CUDA remained uninitialized throughout this independent pass. These checks validate conversion and loader mechanics on Windows/Python 3.12, not downstream task quality or ONNX speed. Full 8,192-token execution and representative capacity/performance remain untested. [ONNX CPU validation results](evidence/onnx-cpu-validation.json). Download this repository at a pinned full Hub commit into `./tokenfold-select-v2`. With the versions above provisioned, the reference API is: ```python from pathlib import Path import sys root = Path("./tokenfold-select-v2").resolve() sys.path.insert(0, str(root / "runtime")) from onnx_score import OnnxScorer scorer = OnnxScorer( root, "267313718e80aa91a56923dea30363fd5c24fde3782c716d6d048891860a2054", development=True, ) response = scorer.score({ "schema_version": 1, "model_revision": "4dbdb5071e7d6316d8eb70dc161b1f2f77f555012f6616d7fa34311188e24cec", "query": "Where is the order being delivered?", "groups": [ {"id": "delivery", "text": "Delivery city: Paris\n"}, {"id": "metadata", "text": "Record created on Monday\n"}, ], }) print(response) # ordered group IDs and scores, not compressed text ``` The adapter verifies the pinned [ONNX manifest](onnx-manifest.json), packs complete original groups with the trained tokenizer, and averages content-token scores back into group scores. Query/separator tokens are not pooled. Protect loader and model files and bound the calling process's lifetime; this is not an OS sandbox. This is not the v1 PEFT adapter or a stock Transformers classification model. ## Run Tokenfold Select v2 with Safetensors Prefer PyTorch tooling? The same trained FP32 parameters are also staged as **`model.safetensors`**: 76 tensors, 98,864,641 values, without a pickle checkpoint. ONNX remains the default for the approved Tokenfold CLI worker. This alternative is a separately provisioned model/reference API, not an automatic CLI backend. This is a **custom joint-group encoder and scalar head**, not a v1 LoRA adapter or stock `AutoModelForSequenceClassification`. Use the included `runtime/safetensors_score.py` with `runtime/joint_group_model.py`, the tokenizer, configuration and manifest. The original source configuration is retained as `source-config.json`; distribution `config.json` identifies the custom model. The canonical architecture file retains its historical prototype docstring; the loader strictly replaces every parameter with the trained weights. The [Safetensors manifest](safetensors-manifest.json) pins all computational files. The CPU reference uses the separately provisioned [pinned PyTorch environment](safetensors-environment-requirements.txt), including a CUDA-enabled Torch build executed strictly on CPU. Other device/build profiles are not validated by this reference and must not inherit its evidence. ```python # Run in a fresh process with caller-owned deadlines; set before importing Torch. import os os.environ["CUDA_VISIBLE_DEVICES"] = "-1" from pathlib import Path import sys root = Path("./tokenfold-select-v2").resolve() sys.path.insert(0, str(root / "runtime")) from safetensors_score import SafetensorsScorer scorer = SafetensorsScorer( root, "bd335629747ba2b437fbd528d9b5b0413523f94f82e7b4b21f4638f2580b9f4e", development=True, ) response = scorer.score({ "schema_version": 1, "model_revision": "4dbdb5071e7d6316d8eb70dc161b1f2f77f555012f6616d7fa34311188e24cec", "query": "Where is the order being delivered?", "groups": [{"id": "delivery", "text": "Delivery city: Paris\n"}], }) print(response) # ordered group scores; allocation remains with your controller ``` **Validated without reconversion:** an independent CPU-only pass confirmed all **76 FP32 tensors / 98,864,641 values match the source checkpoint bit-for-bit**. Both synthetic requests (1 and 8 groups; 20 and 125 tokens) passed the actual Safetensors loader with **zero score difference**, identical rankings and original packing. CUDA remained unavailable and uninitialized. Both owned processes exited cleanly and were waited for. [Safetensors CPU validation results](evidence/safetensors-cpu-validation.json). The earlier conversion run's terminal-report failure remains preserved in the [historical conversion audit](evidence/safetensors-conversion-audit.json); it has not been relabeled successful. These are format/loader mechanics checks, not representative quality, latency or full-8k execution qualification. ## What to know before integrating - **Selection is not a correctness guarantee.** Scores are rankings, not probabilities. Enforce required content, redaction, source order, exact output budgets and deterministic fallback in a trusted controller. A fallback is not a successful model-backed result. - **Bounded means bounded.** The ONNX reference adapter refuses oversized or multi-window inputs before loading the model; it does not truncate or split groups. Research multi-window support is not packaged long-context support. One encoder pass does not mean constant total request work. - **Evidence is still developmental.** Training and evaluation are small and English-centric. Untouched representative task-quality, cold-latency and production qualification remain open. Neither universal superiority nor downstream quality or cost savings is claimed. Keep this model opt-in while evaluating it on your own workload. Do not rely on unqualified compressed context as the sole basis for high-impact decisions. ## Tokenfold Select v2: training, provenance and license Built from [IBM Granite reranker English R2](https://huggingface.co/ibm-granite/granite-embedding-reranker-english-r2/blob/d09d3d6971b689bf9c23839e45a470874d46e13a/README.md), retaining original encoder layers 0–11 and adding group pooling/scoring. Training used 143 TRAIN rows from USGS JSON and CPython code, TRAIN-only head warmup, then three joint FP32 epochs with original-foundation feature distillation. The 87 exposed DEV rows influenced model selection; they are not untouched held-out data. Credit: U.S. Geological Survey earthquake event information and CPython source contributors. No training corpus or evaluation cache is included. Both the ONNX and Safetensors FP32 weights derive from the pinned PyTorch reference; the tokenizer is unchanged. The FP16 timings above describe the earlier PyTorch runtime cast, not the ONNX or Safetensors CPU reference loaders. [Training recipe](TRAINING-RECIPE.json) · [Provenance](provenance.json) · [Change notice and license scope](NOTICE.md) · [Integrity manifest](bundle-manifest.json) **Release status:** ONNX and Safetensors format/loader mechanics checks have passed. This is an opt-in experimental preview, not a quality/performance-qualified GA model. Distributed under [Apache-2.0](LICENSE), with maintainer approval of the recorded model/tokenizer terms and source applicability. That disposition is not a blanket license for unrelated source material or an upstream endorsement. Upstream and CPython license texts are retained under `licenses/`. [Current distribution status](release-status.json).