⚠️ See Disclaimer below before using.

monot5-small-msmarco-10k

A Teradata-compatible Generative Reranker Model

An ONNX-converted, BYOM-ready copy of castorini/monot5-small-msmarco-10k: given a (query, passage) pair, it outputs a single relevance score for that pair. Unlike the other rerankers in this collection, it's a seq2seq model, not a classification head, so it can't run through ONNXEmbeddings -- it's deployed through Teradata's ONNXSeq2Seq instead, using a small graph modification described in "How it works" below.

Reranking in two-stage retrieval

Retrieval pipelines typically run in two stages. The first stage -- a vector search over an index such as HNSW, or classical keyword search -- scans the whole corpus and returns a shortlist (say, the top 20-100 candidates). It's fast because it embeds the query and each document independently and only ever compares the resulting vectors; the two texts never interact. A reranker is a cross-encoder: it reads the query and one candidate together, token by token, and produces a single relevance score for that pair. That joint attention makes it far more accurate than the first-stage vector search, but also too expensive to run over an entire corpus. The standard pattern is therefore to let cheap first-stage retrieval narrow things down, then use a reranker like this one to re-score and re-order just that shortlist.

Overview of this Model

• T5-small, 60M params (ONNX fp32: 294MB)
• 512 max input tokens
• Output: relevance_token_id, a token id that decodes to <bucket_NNN> -- parse NNN and divide by 999 for a [0, 1] relevance score (see Quickstart)
• License: Apache-2.0 (from upstream)
• Reference to Original Model: https://huggingface.co/castorini/monot5-small-msmarco-10k

Quickstart: Deploying this Model in Teradata

import getpass
import teradataml as tdml
from huggingface_hub import hf_hub_download

repo_id  = "martinhillebrandtd/monot5-small-msmarco-10k"
model_id = "monot5-small-msmarco-10k"

# 1. Download artifacts from this repo
hf_hub_download(repo_id=repo_id, filename="onnx/model.onnx", local_dir="./")
hf_hub_download(repo_id=repo_id, filename="tokenizer.json",  local_dir="./")

# 2. Connect to Teradata
tdml.create_context(host=input("host: "), username=input("user: "), password=getpass.getpass("password: "))

# 3. Load model + tokenizer into BYOM tables
tdml.save_byom(model_id=model_id, model_file="onnx/model.onnx", table_name="seq2seq_models")
tdml.save_byom(model_id=model_id, model_file="tokenizer.json",  table_name="seq2seq_tokenizers")

# 4. Score an existing chunks table against one fixed query. No Const_* generation parameters
#    are needed: this graph has no BeamSearch op and no extra inputs beyond
#    input_ids/attention_mask. The bucket number is parsed out of relevance_token_id in SQL,
#    with REGEXP_SUBSTR -- no client-side Python post-processing.
chunks_table = "your_chunks_table"  # columns: chunk_id, chunk_txt
QUERY = "What causes rain?"

query = f"""
SELECT
    chunk_id,
    chunk_txt,
    relevance_token_id,
    CAST(REGEXP_SUBSTR(relevance_token_id, '[0-9]+') AS INTEGER) / 999.0 AS relevance_score
FROM mldb.ONNXSeq2Seq(
    ON (SELECT chunk_id, chunk_txt, 'Query: {QUERY} Document: ' || chunk_txt || ' Relevant:' AS txt
         FROM {chunks_table}) AS InputTable
    ON (SELECT model_id, model FROM seq2seq_models
         WHERE model_id = '{model_id}') AS ModelTable DIMENSION
    ON (SELECT model AS tokenizer FROM seq2seq_tokenizers
         WHERE model_id = '{model_id}') AS TokenizerTable DIMENSION
    USING
        Accumulate('chunk_id', 'chunk_txt')
        ModelOutputTensor('relevance_token_id')
        SkipSpecialTokens('false')
        OverwriteCachedModel('true')
) AS t
"""
result = tdml.DataFrame.from_query(query).to_pandas().sort_values("relevance_score", ascending=False)

Real output, QUERY = "What causes rain?" against three sample chunks:

chunk_id	chunk_txt	relevance_token_id	relevance_score
0	Rain forms when water vapor in clouds condenses into droplets heavy enough to fall.	<bucket_306>	0.306
1	Paris is the capital and most populous city of France.	<bucket_000>	0.0
2	Airplanes generate lift through the shape and angle of their wings.	<bucket_000>	0.0

Combining with vector search for two-stage retrieval

A typical pipeline: embed your corpus once with an embedding model, then at query time use VectorDistance to get a shortlist of candidates by cosine distance, and rerank only that shortlist with this model.

-- Stage 1: shortlist the 20 nearest chunks to the query embedding
CREATE VOLATILE TABLE shortlist AS (
    SELECT ref_id AS chunk_id, distance
    FROM TD_VECTORDISTANCE(
        ON query_embedding AS TargetTable
        ON chunk_embeddings AS ReferenceTable DIMENSION
        USING
            TargetIDColumn('query_id')
            TargetFeatureColumns('[emb_0:emb_383]')
            RefIDColumn('chunk_id')
            RefFeatureColumns('[emb_0:emb_383]')
            DistanceMeasure('cosine')
            TopK(20)
    ) AS dt
) WITH DATA ON COMMIT PRESERVE ROWS;

-- Stage 2: rerank only those 20 shortlisted chunks (see Quickstart above, with
-- your_chunks_table replaced by a join against `shortlist`)

How it works

Teradata's ONNXSeq2Seq is built around com.microsoft.BeamSearch and always returns tokenizer-decoded text, never a raw number -- it's designed to reuse a text-generation model as a generic "text in, next-token-as-text out" engine. The original model reads relevance as the softmax probability of the token "true" at the first decode step -- a continuous score, not free text. Run unmodified, it would only ever return the literal string "true" or "false", discarding exactly the information a reranker needs.

This repo's ONNX graph is modified (graph surgery, no beam search, no third-party conversion package) to smuggle that score through as decoded text instead:

1. 1000 dedicated vocabulary tokens, <bucket_000>...<bucket_999>, are added to the tokenizer -- 3-decimal-digit precision, i.e. quantization error ≤ 0.0005. They're only ever used as an output lookup, never fed through the model's own embedding layer or lm_head, so no model embedding resizing is needed.
2. A small wrapper module runs the real T5 encoder + one real decoder step, takes the softmax over just the true/false logits to get the real probability, quantizes it to a bucket index, and outputs that bucket token's id directly, as an int64 tensor named relevance_token_id. The exported graph has zero BeamSearch nodes -- just a plain feedforward computation.
3. ONNXSeq2Seq's ModelOutputTensor argument validates against the model's actual declared output tensors, not a hardcoded whitelist -- so relevance_token_id is accepted, and its value is looked up in the tokenizer's vocabulary and returned as "<bucket_NNN>".

Verified end-to-end, both locally (onnxruntime) and against a live Vantage instance, versus the true PyTorch score (T5ForConditionalGeneration, real softmax over the true/false logits):

	mean |delta|	max |delta|
Local ONNX (bucket-decoded) vs. true PyTorch	0.00021	0.00041
Live Teradata ONNXSeq2Seq vs. true PyTorch	0.00021	0.00041

Both at the quantization noise floor for 1000 buckets (max possible error ≈ 1/999 ≈ 0.001).

How this model was converted

convert.py is fully self-contained: no third-party conversion package is used. It (1) adds the 1000 bucket tokens to the tokenizer, (2) wraps T5ForConditionalGeneration's real forward pass in a small torch.nn.Module that computes the softmax score and looks up the matching bucket token id, and (3) exports via plain torch.onnx.export -- no merged/BeamSearch export machinery needed. See convert.py for the full implementation, and test_local.py for the ONNX-vs-PyTorch parity check above. The same technique generalizes to other seq2seq "generative" rerankers that score by reading a token probability rather than a classification logit -- swap in the target model's own true/false (or equivalent) token ids and prompt template.

---

Disclaimer

DISCLAIMER: The content herein ("Content") is provided "AS IS" and is not covered by any Teradata Operations, Inc. and its affiliates ("Teradata") agreements. Its listing here does not constitute certification or endorsement by Teradata.

To the extent any of the Content contains or is related to any artificial intelligence ("AI") or other language learning models ("Models") that interoperate with the products and services of Teradata, by accessing, bringing, deploying or using such Models, you acknowledge and agree that you are solely responsible for ensuring compliance with all applicable laws, regulations, and restrictions governing the use, deployment, and distribution of AI technologies. This includes, but is not limited to, AI Diffusion Rules, European Union AI Act, AI-related laws and regulations, privacy laws, export controls, and financial or sector-specific regulations.

While Teradata may provide support, guidance, or assistance in the deployment or implementation of Models to interoperate with Teradata's products and/or services, you remain fully responsible for ensuring that your Models, data, and applications comply with all relevant legal and regulatory obligations. Our assistance does not constitute legal or regulatory approval, and Teradata disclaims any liability arising from non-compliance with applicable laws.

You must determine the suitability of the Models for any purpose. Given the probabilistic nature of machine learning and modeling, the use of the Models may in some situations result in incorrect output that does not accurately reflect the action generated. You should evaluate the accuracy of any output as appropriate for your use case, including by using human review of the output.