docs/INFERENCE.md

Using Pivot

Pivot evaluates a supplied set of options for each context. It is a bidirectional encoder with a set scorer, so its answer is an index into your options, not generated text. Use 2–16 distinct candidate meanings for the structured native interface. A single plain choose() call accepts two or more options; large candidate lists cost more memory and time.

Load once

import torch
from transformers import AutoModel, AutoTokenizer

repo = "Q1z/Pivot"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval()

This repository ships custom Transformers code. Inspect it before loading with trust_remote_code=True in a production environment. A CPU-only PyTorch build works. For CUDA, call model.to("cuda"). Keep the tokenizer and model loaded across requests.

Plain choice

context = "CONTEXT:\nA user requests a refund for an incorrect charge."
options = ["route to billing", "route to technical support", "route to sales"]
result = model.choose(tokenizer, context, options)
assert result["choice"] == options[result["index"]]
assert len(result["probs"]) == len(options)

The probability vector follows the original option order and sums to approximately one. Changing the option wording or adding an option changes the set being scored. The public benchmark uses the CONTEXT: prefix, a 512-token context limit and a 128-token option limit.

Native structured choice

decide_native() accepts a nonempty context and 2–16 objects with unique id and distinct text values. Optional kind: "abstain" is allowed on at most one candidate. Machine IDs stay outside the option encoder.

result = model.decide_native(
    tokenizer,
    "A customer disputes an invoice.",
    [
        {"id": "billing", "text": "route to billing"},
        {"id": "technical", "text": "route to technical support"},
        {"id": "unknown", "text": "none of the routes fits", "kind": "abstain"},
    ],
)
print(result["selected"]["id"], result["prob_vector"])

The abstain option competes with the other supplied options; it is not an independently calibrated refusal threshold.

Several typed decisions

result = model.decide(tokenizer, "Customer invoice disputed", [
    {"id": "route", "primitive": "choice", "options": ["billing", "tech", "sales"]},
    {"id": "approve", "primitive": "noul", "options": ["true", "false"]},
    {"id": "severity", "primitive": "score", "options": ["0", "1", "2", "3"]},
])

The typed method scores each question in turn. The examples under serving/ show request and response shapes.

Cache repeated candidates

For repeated contexts against an unchanged option set:

options = ["billing support", "technical support", "sales"]
candidate_vectors = model.encode_candidates(tokenizer, options)
context_vector = model.encode_context(tokenizer, "Invoice is incorrect")
result = model.choose_cached(context_vector, candidate_vectors, options)

Cache embeddings only with the same model, tokenizer, precision and option text. This path can avoid repeated option encoding. Published speed measurements instead use the uncached end-to-end path. Runnable examples.