MultiSense/SaleIntent_bert

SaleIntent-BERT

Our work has been acepted by EMNLP 2026

SaleIntent-BERT is a fine-tuned BERT classifier that reads a complete sales conversation and predicts how the customer ended up — from clear purchase intent down to hostility. It is the outcome-scoring half of the SalesLLM benchmark.

Scoring a sales dialogue with an LLM judge alone conflates two different questions: did the salesperson run a good process? and did the customer actually want to buy at the end? A model can be articulate, polite, and well-structured while the customer walks away — and an LLM judge, reading the whole transcript, tends to reward the articulate process. SaleIntent-BERT answers the second question independently, by looking only at where the conversation landed.

It reaches 93.51% accuracy on Chinese and 92.94% on English, and pairs with the LLM judge to produce the final SalesLLM Score.

Task	5-class sequence classification over a full dialogue
Base model	BERT (see config.json for the exact checkpoint)
Languages	Chinese, English
Input	Flattened multi-turn dialogue, last 128 tokens
Accuracy	93.51% (ZH), 92.94% (EN)
License	Apache 2.0

---

Labels

The five classes are an outcome grade, not a monotonic intent ladder. Each maps to a point score used in the final benchmark metric:

Label	Meaning	Score
A	Customer has clear purchase intent	10
B	Customer possibly has intent	8
C	Customer has no purchase intent	6
X	Customer has weak intent; dismissive / going through the motions	4
F	Customer is abusive or complaining	2

Read the ordering carefully — C (no intent) scores higher than X (weak, dismissive). A clean, honest "no" is a better conversational outcome than one the salesperson dragged into disengaged stonewalling, and an outright hostile ending (F) is worst of all. The scale grades the state the salesperson left the customer in, not just how close the sale was.

Class indices are A=0, B=1, C=2, F=3, X=4. Do not assume index order matches score order — always resolve through model.config.id2label.

---

Input Format

The model expects the dialogue flattened into a single string with explicit speaker tags, then tail-truncated to the last 128 tokens:

def flatten_dialogue(messages):
    out = ""
    for msg in messages:
        tag = "[ASSISTANT]" if msg["role"] == "assistant" else "[USER]"
        out += tag + msg["content"]
    return out

Two details are load-bearing:

• Speaker tags, no separators. [ASSISTANT]/[USER] are concatenated directly against the message text with no spaces or newlines. The model was trained on exactly this string shape.
• Tail truncation, not head. Buying intent is decided at the end of a conversation, so the last 128 tokens are kept and everything before is dropped. Standard truncation=True keeps the head and will silently feed the model the opening pleasantries instead of the outcome — this is the single most common way to get bad predictions from this model.

last_token_num=128 is the benchmark's validated setting. The window is deliberately short: a longer window pulls in mid-conversation negotiation that dilutes the end-state signal.

---

Usage

Direct inference

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "MultiSense/SaleIntent_bert"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()

LABEL2SCORE = {"A": 10, "B": 8, "C": 6, "X": 4, "F": 2}
N, MAX_LEN = 128, 512

def flatten_dialogue(messages):
    return "".join(
        ("[ASSISTANT]" if m["role"] == "assistant" else "[USER]") + m["content"]
        for m in messages
    )

def encode_tail(text, n=N, max_length=MAX_LEN):
    """Keep the LAST n tokens — intent lives at the end of the dialogue."""
    toks = tok.tokenize(text)[-min(n, max_length - 2):]
    ids = [tok.cls_token_id] + tok.convert_tokens_to_ids(toks) + [tok.sep_token_id]
    mask = [1] * len(ids)
    pad = max_length - len(ids)
    return {
        "input_ids": torch.tensor([ids + [tok.pad_token_id] * pad]),
        "attention_mask": torch.tensor([mask + [0] * pad]),
    }

messages = [
    {"role": "user",      "content": "你好，我想了解一下你们的降噪耳机。"},
    {"role": "assistant", "content": "好的，这款支持32dB混合降噪，续航38小时，售价1999元。"},
    {"role": "user",      "content": "听起来不错，那我下单一个吧。"},
]

with torch.no_grad():
    logits = model(**encode_tail(flatten_dialogue(messages))).logits

label = model.config.id2label[logits.argmax(-1).item()]
print(label, LABEL2SCORE[label])   # -> A 10

As part of the SalesLLM score

comprehensive_score.py runs this model over a results file and blends it with the LLM judge:

python salesllm/comprehensive_score.py \
  --bert_path "MultiSense/SaleIntent_bert" \
  --source_file "./results/zh/<output>.jsonl" \
  --llm_model_name "<judge_model>" \
  --api_key "<key>" --end_point "<base_url>" \
  --last_token_num 128 \
  --proportion 0.6

The final score is a weighted blend of the two signals:

final_score = proportion * LABEL2SCORE[bert_label] + (1 - proportion) * llm_judge_score

The benchmark uses proportion = 0.6 — outcome weighted slightly above process, because outcome is the harder signal to game. Both components are on the same 0–10 scale, so the blend is directly interpretable.

Output is written to <source_file>_scored.json, with A/B (the predicted label), conversation_quality (the LLM judge's 0–10), and final_score added to each record.

---

Evaluation

Language	Accuracy
Chinese	93.51%
English	92.94%

The combined pipeline (this classifier at 0.6 + LLM judge at 0.4) achieves a Pearson correlation of r = 0.98 with human ratings of overall sales performance, which is the result that justifies using the automated score in place of human annotation at benchmark scale.

Accuracy is reported over the full 5-class problem. Note that the classes are not balanced in realistic sales data — successful closes are rarer than non-purchases, and F (abusive) is rarest of all — so per-class recall on the tail classes will be lower than the aggregate figure suggests. If your use case hinges on detecting F or X specifically, measure per-class performance on your own data before relying on it.

---

Limitations and Risks

• Truncation is the main failure mode. Feed it head-truncated text and predictions degrade badly while still looking confident. Always tail-truncate.
• 128-token window. Intent expressed early and never restated near the end will be missed. Conversations that end with an off-topic exchange can also mislead it.
• Trained on simulated + real sales dialogue in Financial Services and Consumer Goods. Other verticals, other conversation formats (email threads, support tickets), and non-sales dialogue are out of distribution.
• Format-coupled. The [ASSISTANT]/[USER] tagging is part of the learned input representation, not a cosmetic choice. Different tags or added whitespace will shift predictions.
• Not a purchase predictor. It classifies expressed intent at the end of a conversation. Stated intent is not a real-world conversion rate, and it should not be used to forecast revenue or to score individual human salespeople for performance management.
• Ordinal scores are a benchmark convention. The 10/8/6/4/2 mapping was chosen for the SalesLLM metric. The intervals are not calibrated probabilities and should not be treated as such.
• Inherited bias. Predictions may vary with dialect, phrasing formality, and translationese in ways that correlate with demographics. Do not use it to gate access, rank customers, or make decisions affecting individuals.

---

Citation

@misc{su2026sellmoreplayless,
      title={Sell More, Play Less: Benchmarking LLM Realistic Selling Skill}, 
      author={Xuanbo Su and Wenhao Hu and Le Zhan and Yuting Xie and Kailin Lyu and Kaijie Chen and Ziwei Li and Yeqiang Wang and Haibo Su and Yunzhang Chen and Ling Huang},
      year={2026},
      eprint={2604.07054},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2604.07054}, 
}

Related

• 📖 Paper
• 📊 SalesLLM benchmark & code
• 🤗 CustomerLM — the user simulator that generates the customer side of the dialogues this model scores