| # Using Pivot |
|
|
| Pivot evaluates a **supplied set of options** for each context. It is a bidirectional encoder with a set scorer, so its answer is an index into your options, not generated text. Use 2–16 distinct candidate meanings for the structured native interface. A single plain `choose()` call accepts two or more options; large candidate lists cost more memory and time. |
|
|
| ## Load once |
|
|
| ```python |
| import torch |
| from transformers import AutoModel, AutoTokenizer |
| |
| repo = "Q1z/Pivot" |
| tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True) |
| model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval() |
| ``` |
|
|
| This repository ships custom Transformers code. Inspect it before loading with `trust_remote_code=True` in a production environment. A CPU-only PyTorch build works. For CUDA, call `model.to("cuda")`. Keep the tokenizer and model loaded across requests. |
|
|
| ## Plain choice |
|
|
| ```python |
| context = "CONTEXT:\nA user requests a refund for an incorrect charge." |
| options = ["route to billing", "route to technical support", "route to sales"] |
| result = model.choose(tokenizer, context, options) |
| assert result["choice"] == options[result["index"]] |
| assert len(result["probs"]) == len(options) |
| ``` |
|
|
| The probability vector follows the original option order and sums to approximately one. Changing the option wording or adding an option changes the set being scored. The public benchmark uses the `CONTEXT:` prefix, a 512-token context limit and a 128-token option limit. |
|
|
| ## Native structured choice |
|
|
| `decide_native()` accepts a nonempty context and 2–16 objects with unique `id` and distinct `text` values. Optional `kind: "abstain"` is allowed on at most one candidate. Machine IDs stay outside the option encoder. |
|
|
| ```python |
| result = model.decide_native( |
| tokenizer, |
| "A customer disputes an invoice.", |
| [ |
| {"id": "billing", "text": "route to billing"}, |
| {"id": "technical", "text": "route to technical support"}, |
| {"id": "unknown", "text": "none of the routes fits", "kind": "abstain"}, |
| ], |
| ) |
| print(result["selected"]["id"], result["prob_vector"]) |
| ``` |
|
|
| The abstain option competes with the other supplied options; it is not an independently calibrated refusal threshold. |
|
|
| ## Several typed decisions |
|
|
| ```python |
| result = model.decide(tokenizer, "Customer invoice disputed", [ |
| {"id": "route", "primitive": "choice", "options": ["billing", "tech", "sales"]}, |
| {"id": "approve", "primitive": "noul", "options": ["true", "false"]}, |
| {"id": "severity", "primitive": "score", "options": ["0", "1", "2", "3"]}, |
| ]) |
| ``` |
|
|
| The typed method scores each question in turn. The examples under [`serving/`](../serving/) show request and response shapes. |
|
|
| ## Cache repeated candidates |
|
|
| For repeated contexts against an unchanged option set: |
|
|
| ```python |
| options = ["billing support", "technical support", "sales"] |
| candidate_vectors = model.encode_candidates(tokenizer, options) |
| context_vector = model.encode_context(tokenizer, "Invoice is incorrect") |
| result = model.choose_cached(context_vector, candidate_vectors, options) |
| ``` |
|
|
| Cache embeddings only with the same model, tokenizer, precision and option text. This path can avoid repeated option encoding. Published speed measurements instead use the uncached end-to-end path. [Runnable examples](../examples/cached_candidates.py). |
|
|
