| ---
|
| language:
|
| - en
|
| base_model: ibm-granite/granite-embedding-reranker-english-r2
|
| base_model_relation: finetune
|
| license: apache-2.0
|
| library_name: onnxruntime
|
| tags:
|
| - tokenfold
|
| - context-compression
|
| - extractive-selection
|
| - experimental
|
| - custom-model
|
| - onnx
|
| - safetensors
|
| ---
|
|
|
| # Tokenfold Select v2
|
|
|
| ### Build smaller contexts. Keep the original words.
|
|
|
| **Query-aware context selection in a compact, 98.9M-parameter model: ONNX by default, Safetensors for PyTorch workflows.**
|
|
|
| LLM workflows collect more context than every request needs. Tokenfold Select v2
|
| helps your application choose what to keep: it scores text groups together with
|
| the query, then hands those scores to an allocator that assembles a smaller
|
| context from the original text. No generated summaries or paraphrases.
|
|
|
| **Experimental ONNX + Safetensors preview · English · Local inference**
|
|
|
| ## Why Tokenfold Select v2?
|
|
|
| - **Score groups together.** One joint encoder pass scores all groups in an
|
| admitted single-window request, rather than encoding each query/span pair
|
| separately.
|
| - **Keep the source intact where it counts.** The extractive workflow retains
|
| selected original groups; your allocator controls required content, source
|
| order and the final token budget.
|
| - **Use a compact specialist.** A trained 12-layer encoder with a shared group
|
| scoring head—not a generative LLM or the v1 LoRA adapter.
|
| - **Run locally.** Offline-capable inference with caller-provisioned weights and
|
| explicit precision settings. No hosted model service is required.
|
| - **Inspect every selection.** Ordered group scores give your application a
|
| clear interface for ranking, allocation and debugging.
|
|
|
| ```text
|
| Query + original text groups
|
| ↓
|
| Select v2 group scores
|
| ↓
|
| Allocator + required-content rules + token budget
|
| ↓
|
| Selected context, in original wording
|
| ```
|
|
|
| For developers exploring context selection in retrieval, tool/API responses,
|
| coding assistants and agent histories. The model returns **scores**, not a
|
| compressed document; grouping and budget enforcement remain in your application.
|
|
|
| ## Tokenfold Select v2: reference performance
|
|
|
| Using the PyTorch/CUDA FP16 reference runtime (not ONNX) on the JSON development
|
| fixture at 50% retention, Tokenfold Select v2 recorded **150 ms
|
| warm p50 compression latency**, versus **208 ms for Select v1**—about **28% lower**.
|
|
|
| The table shows **p50 / p95 milliseconds** at the same retention target:
|
|
|
| | Development fixture | Tokenfold Select v2 **FP16** | Tokenfold Select v1 FP32 | Headroom Kompress v2 FP32 |
|
| | --- | ---: | ---: | ---: |
|
| | JSON | **150.199 / 161.996** | 207.644 / 215.143 | Budget refusal: 0/6 completions |
|
| | Code | **268.713 / 282.441** | 451.916 / 466.936 | Budget refusal: 0/6 completions |
|
| | Logs | **1842.744 / 1862.359** | 3760.490 / 3788.029 | Budget refusal: 0/6 completions |
|
| | Multi-turn | 267.051 / 271.358 | 563.040 / 568.882 | **94.577 / 97.534** |
|
| | Retrieval | **773.447 / 803.698** | 2046.032 / 2070.185 | 2733.641 / 2751.180 |
|
|
|
| **Scope matters:** one exposed source per family, six repeated requests per arm
|
| and budget, on one local RTX 5080 using the PyTorch/CUDA reference runtime, not ONNX. These are empirical warm development results,
|
| not independent-source confidence bounds or proof of downstream answer quality.
|
| Kompress is faster on multi-turn where both complete; its budget refusals are
|
| failures, not evidence of a successful-compression speedup for Select v2.
|
|
|
| The complete comparison covered **180/180 requests** across two retention targets:
|
| 132 valid compressions and 48 healthy Kompress budget refusals, with no omissions.
|
| Refusals retain infinite completion latency in the failure-aware summaries.
|
| The clock includes fresh grouping, tokenization, scoring, native allocation/IPC
|
| and output recount; it excludes loading, warmup and file I/O. This measures the
|
| standalone Kompress compressor, not the full Headroom pipeline.
|
|
|
| [See all workload/budget results and methodology](evidence/development-compression.json).
|
|
|
| ## Run Tokenfold Select v2 with ONNX
|
|
|
| The staged model is a **FP32 ONNX graph plus its external weights**:
|
| `model.onnx` and `model.onnx.data`. Keep both files together. No pickle-based `.pt` checkpoint
|
| is included in this upload candidate. The optional Safetensors file is separate;
|
| ONNX inference does not require PyTorch.
|
|
|
| | Component | Current scope |
|
| | --- | --- |
|
| | Model | 98,864,641 parameters, 12 encoder layers, ONNX opset 18 |
|
| | Graph IO | int64 `input_ids` / `attention_mask`; FP32 `token_scores`, shape `[1, tokens]` |
|
| | Admission | One complete window; at most 8,192 encoded tokens and 512 groups |
|
| | Runtime | ONNX Runtime, explicit CPUExecutionProvider; no implicit provider fallback |
|
| | Software | Python 3.12 and the [pinned ONNX environment](environment-requirements.txt) |
|
|
|
| The ONNX graph and reference adapter passed **6/6 synthetic CPU parity checks**,
|
| including two real packaged-loader requests and dynamic sequence lengths up to
|
| 2,048 tokens / 512 pooled group ranges. Maximum absolute score error was
|
| **0.0000878**, below the predeclared 0.0001 tolerance; ranking agreement was
|
| approximately 1.0. CUDA remained uninitialized throughout this independent pass.
|
|
|
| These checks validate conversion and loader mechanics on Windows/Python 3.12,
|
| not downstream task quality or ONNX speed. Full 8,192-token execution and
|
| representative capacity/performance remain untested.
|
| [ONNX CPU validation results](evidence/onnx-cpu-validation.json).
|
|
|
| Download this repository at a pinned full Hub commit into `./tokenfold-select-v2`.
|
| With the versions above provisioned, the reference API is:
|
|
|
| ```python
|
| from pathlib import Path
|
| import sys
|
|
|
| root = Path("./tokenfold-select-v2").resolve()
|
| sys.path.insert(0, str(root / "runtime"))
|
| from onnx_score import OnnxScorer
|
|
|
| scorer = OnnxScorer(
|
| root,
|
| "267313718e80aa91a56923dea30363fd5c24fde3782c716d6d048891860a2054",
|
| development=True,
|
| )
|
| response = scorer.score({
|
| "schema_version": 1,
|
| "model_revision": "4dbdb5071e7d6316d8eb70dc161b1f2f77f555012f6616d7fa34311188e24cec",
|
| "query": "Where is the order being delivered?",
|
| "groups": [
|
| {"id": "delivery", "text": "Delivery city: Paris\n"},
|
| {"id": "metadata", "text": "Record created on Monday\n"},
|
| ],
|
| })
|
| print(response) # ordered group IDs and scores, not compressed text
|
| ```
|
|
|
| The adapter verifies the pinned [ONNX manifest](onnx-manifest.json), packs complete
|
| original groups with the trained tokenizer, and averages content-token scores
|
| back into group scores. Query/separator tokens are not pooled. Protect loader and
|
| model files and bound the calling process's lifetime; this is not an OS sandbox.
|
| This is not the v1 PEFT adapter or a stock Transformers classification model.
|
|
|
| ## Run Tokenfold Select v2 with Safetensors
|
|
|
| Prefer PyTorch tooling? The same trained FP32 parameters are also staged as
|
| **`model.safetensors`**: 76 tensors, 98,864,641 values, without a pickle checkpoint.
|
| ONNX remains the default for the approved Tokenfold CLI worker. This alternative
|
| is a separately provisioned model/reference API, not an automatic CLI backend.
|
|
|
| This is a **custom joint-group encoder and scalar head**, not a v1 LoRA adapter
|
| or stock `AutoModelForSequenceClassification`. Use the included
|
| `runtime/safetensors_score.py` with `runtime/joint_group_model.py`, the tokenizer,
|
| configuration and manifest. The original source configuration is retained as
|
| `source-config.json`; distribution `config.json` identifies the custom model.
|
| The canonical architecture file retains its historical prototype docstring;
|
| the loader strictly replaces every parameter with the trained weights.
|
|
|
| The [Safetensors manifest](safetensors-manifest.json) pins all computational
|
| files. The CPU reference uses the separately provisioned
|
| [pinned PyTorch environment](safetensors-environment-requirements.txt), including
|
| a CUDA-enabled Torch build executed strictly on CPU. Other device/build profiles
|
| are not validated by this reference and must not inherit its evidence.
|
|
|
| ```python
|
| # Run in a fresh process with caller-owned deadlines; set before importing Torch.
|
| import os
|
| os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
|
| from pathlib import Path
|
| import sys
|
|
|
| root = Path("./tokenfold-select-v2").resolve()
|
| sys.path.insert(0, str(root / "runtime"))
|
| from safetensors_score import SafetensorsScorer
|
|
|
| scorer = SafetensorsScorer(
|
| root,
|
| "bd335629747ba2b437fbd528d9b5b0413523f94f82e7b4b21f4638f2580b9f4e",
|
| development=True,
|
| )
|
| response = scorer.score({
|
| "schema_version": 1,
|
| "model_revision": "4dbdb5071e7d6316d8eb70dc161b1f2f77f555012f6616d7fa34311188e24cec",
|
| "query": "Where is the order being delivered?",
|
| "groups": [{"id": "delivery", "text": "Delivery city: Paris\n"}],
|
| })
|
| print(response) # ordered group scores; allocation remains with your controller
|
| ```
|
|
|
| **Validated without reconversion:** an independent CPU-only pass confirmed
|
| all **76 FP32 tensors / 98,864,641 values match the source checkpoint bit-for-bit**.
|
| Both synthetic requests (1 and 8 groups; 20 and 125 tokens) passed the actual
|
| Safetensors loader with **zero score difference**, identical rankings and original
|
| packing. CUDA remained unavailable and uninitialized. Both owned processes exited
|
| cleanly and were waited for.
|
|
|
| [Safetensors CPU validation results](evidence/safetensors-cpu-validation.json).
|
| The earlier conversion run's terminal-report failure remains preserved in the
|
| [historical conversion audit](evidence/safetensors-conversion-audit.json); it has
|
| not been relabeled successful. These are format/loader mechanics checks, not
|
| representative quality, latency or full-8k execution qualification.
|
|
|
| ## What to know before integrating
|
|
|
| - **Selection is not a correctness guarantee.** Scores are rankings, not
|
| probabilities. Enforce required content, redaction, source order, exact output
|
| budgets and deterministic fallback in a trusted controller. A fallback is not
|
| a successful model-backed result.
|
| - **Bounded means bounded.** The ONNX reference adapter refuses oversized or
|
| multi-window inputs before loading the model; it does not truncate or split
|
| groups. Research multi-window support is not packaged long-context support.
|
| One encoder pass does not mean constant total request work.
|
| - **Evidence is still developmental.** Training and evaluation are small and
|
| English-centric. Untouched representative task-quality, cold-latency and
|
| production qualification remain open. Neither universal superiority nor
|
| downstream quality or cost savings is claimed.
|
|
|
| Keep this model opt-in while evaluating it on your own workload. Do not rely on
|
| unqualified compressed context as the sole basis for high-impact decisions.
|
|
|
| ## Tokenfold Select v2: training, provenance and license
|
|
|
| Built from [IBM Granite reranker English R2](https://huggingface.co/ibm-granite/granite-embedding-reranker-english-r2/blob/d09d3d6971b689bf9c23839e45a470874d46e13a/README.md),
|
| retaining original encoder layers 0–11 and adding group pooling/scoring. Training
|
| used 143 TRAIN rows from USGS JSON and CPython code, TRAIN-only head warmup, then
|
| three joint FP32 epochs with original-foundation feature distillation. The 87
|
| exposed DEV rows influenced model selection; they are not untouched held-out data.
|
|
|
| Credit: U.S. Geological Survey earthquake event information and CPython source
|
| contributors. No training corpus or evaluation cache is included. Both the ONNX and Safetensors FP32 weights derive from the pinned PyTorch reference; the
|
| tokenizer is unchanged. The FP16 timings above describe the earlier PyTorch
|
| runtime cast, not the ONNX or Safetensors CPU reference loaders.
|
|
|
| [Training recipe](TRAINING-RECIPE.json) · [Provenance](provenance.json) ·
|
| [Change notice and license scope](NOTICE.md) · [Integrity manifest](bundle-manifest.json)
|
|
|
| **Release status:** ONNX and Safetensors format/loader mechanics checks have passed.
|
| This is an opt-in experimental preview, not a quality/performance-qualified GA
|
| model. Distributed under [Apache-2.0](LICENSE), with maintainer approval of the
|
| recorded model/tokenizer terms and source applicability. That disposition is not
|
| a blanket license for unrelated source material or an upstream endorsement.
|
| Upstream and CPython license texts are retained under `licenses/`.
|
| [Current distribution status](release-status.json).
|
|
|
