| --- |
| language: |
| - zh |
| - en |
| library_name: transformers |
| license: apache-2.0 |
| pipeline_tag: text-classification |
| tags: |
| - bert |
| - text-classification |
| - sales |
| - intent-classification |
| - dialogue |
| - evaluation |
| --- |
| |
| # SaleIntent-BERT |
|
|
| **Paper**: [Sell More, Play Less: Benchmarking LLM Realistic Selling Skill](https://huggingface.co/papers/2604.07054) |
|
|
| **SaleIntent-BERT** is a fine-tuned BERT classifier that reads a complete sales conversation and predicts **how the customer ended up** — from clear purchase intent down to hostility. It is the outcome-scoring half of the [SalesLLM benchmark](https://github.com/Bairong-Xdynamics/Benchmarking-LLM-Realistic-Selling-Skill). |
|
|
| Scoring a sales dialogue with an LLM judge alone conflates two different questions: *did the salesperson run a good process?* and *did the customer actually want to buy at the end?* A model can be articulate, polite, and well-structured while the customer walks away — and an LLM judge, reading the whole transcript, tends to reward the articulate process. SaleIntent-BERT answers the second question independently, by looking only at where the conversation landed. |
|
|
| It reaches **93.51% accuracy on Chinese and 92.94% on English**, and pairs with the LLM judge to produce the final SalesLLM Score. |
|
|
| | | | |
| | :--- | :--- | |
| | **Task** | 5-class sequence classification over a full dialogue | |
| | **Base model** | BERT (see `config.json` for the exact checkpoint) | |
| | **Languages** | Chinese, English | |
| | **Input** | Flattened multi-turn dialogue, **last 128 tokens** | |
| | **Accuracy** | 93.51% (ZH), 92.94% (EN) | |
| | **License** | Apache 2.0 | |
|
|
| --- |
|
|
| ## Labels |
|
|
| The five classes are an **outcome grade**, not a monotonic intent ladder. Each maps to a point score used in the final benchmark metric: |
|
|
| | Label | Meaning | Score | |
| | :---: | :--- | :---: | |
| | **A** | Customer has **clear** purchase intent | 10 | |
| | **B** | Customer **possibly** has intent | 8 | |
| | **C** | Customer has **no** purchase intent | 6 | |
| | **X** | Customer has **weak** intent; dismissive / going through the motions | 4 | |
| | **F** | Customer is **abusive or complaining** | 2 | |
|
|
| Read the ordering carefully — **C (no intent) scores higher than X (weak, dismissive)**. A clean, honest "no" is a better conversational outcome than one the salesperson dragged into disengaged stonewalling, and an outright hostile ending (F) is worst of all. The scale grades the *state the salesperson left the customer in*, not just how close the sale was. |
|
|
| Class indices are `A=0, B=1, C=2, F=3, X=4`. Do not assume index order matches score order — always resolve through `model.config.id2label`. |
|
|
| --- |
|
|
| ## Input Format |
|
|
| The model expects the dialogue **flattened into a single string** with explicit speaker tags, then **tail-truncated to the last 128 tokens**: |
|
|
| ```python |
| def flatten_dialogue(messages): |
| out = "" |
| for msg in messages: |
| tag = "[ASSISTANT]" if msg["role"] == "assistant" else "[USER]" |
| out += tag + msg["content"] |
| return out |
| ``` |
|
|
| Two details are load-bearing: |
|
|
| - **Speaker tags, no separators.** `[ASSISTANT]`/`[USER]` are concatenated directly against the message text with no spaces or newlines. The model was trained on exactly this string shape. |
| - **Tail truncation, not head.** Buying intent is decided at the *end* of a conversation, so the last 128 tokens are kept and everything before is dropped. Standard `truncation=True` keeps the *head* and will silently feed the model the opening pleasantries instead of the outcome — this is the single most common way to get bad predictions from this model. |
|
|
| `last_token_num=128` is the benchmark's validated setting. The window is deliberately short: a longer window pulls in mid-conversation negotiation that dilutes the end-state signal. |
|
|
| --- |
|
|
| ## Usage |
|
|
| ### Direct inference |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| |
| model_id = "MultiSense/SaleIntent_bert" |
| tok = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForSequenceClassification.from_pretrained(model_id).eval() |
| |
| LABEL2SCORE = {"A": 10, "B": 8, "C": 6, "X": 4, "F": 2} |
| N, MAX_LEN = 128, 512 |
| |
| def flatten_dialogue(messages): |
| return "".join( |
| ("[ASSISTANT]" if m["role"] == "assistant" else "[USER]") + m["content"] |
| for m in messages |
| ) |
| |
| def encode_tail(text, n=N, max_length=MAX_LEN): |
| """Keep the LAST n tokens — intent lives at the end of the dialogue.""" |
| toks = tok.tokenize(text)[-min(n, max_length - 2):] |
| ids = [tok.cls_token_id] + tok.convert_tokens_to_ids(toks) + [tok.sep_token_id] |
| mask = [1] * len(ids) |
| pad = max_length - len(ids) |
| return { |
| "input_ids": torch.tensor([ids + [tok.pad_token_id] * pad]), |
| "attention_mask": torch.tensor([mask + [0] * pad]), |
| } |
| |
| messages = [ |
| {"role": "user", "content": "你好，我想了解一下你们的降噪耳机。"}, |
| {"role": "assistant", "content": "好的，这款支持32dB混合降噪，续航38小时，售价1999元。"}, |
| {"role": "user", "content": "听起来不错，那我下单一个吧。"}, |
| ] |
| |
| with torch.no_grad(): |
| logits = model(**encode_tail(flatten_dialogue(messages))).logits |
| |
| label = model.config.id2label[logits.argmax(-1).item()] |
| print(label, LABEL2SCORE[label]) # -> A 10 |
| ``` |
|
|
| ### As part of the SalesLLM score |
|
|
| `comprehensive_score.py` runs this model over a results file and blends it with the LLM judge: |
|
|
| ```bash |
| python salesllm/comprehensive_score.py \ |
| --bert_path "MultiSense/SaleIntent_bert" \ |
| --source_file "./results/zh/<output>.jsonl" \ |
| --llm_model_name "<judge_model>" \ |
| --api_key "<key>" --end_point "<base_url>" \ |
| --last_token_num 128 \ |
| --proportion 0.6 |
| ``` |
|
|
| The final score is a weighted blend of the two signals: |
|
|
| ``` |
| final_score = proportion * LABEL2SCORE[bert_label] + (1 - proportion) * llm_judge_score |
| ``` |
|
|
| The benchmark uses **`proportion = 0.6`** — outcome weighted slightly above process, because outcome is the harder signal to game. Both components are on the same 0–10 scale, so the blend is directly interpretable. |
|
|
| Output is written to `<source_file>_scored.json`, with `A/B` (the predicted label), `conversation_quality` (the LLM judge's 0–10), and `final_score` added to each record. |
|
|
| --- |
|
|
| ## Evaluation |
|
|
| | Language | Accuracy | |
| | :--- | :---: | |
| | Chinese | **93.51%** | |
| | English | **92.94%** | |
|
|
| The combined pipeline (this classifier at 0.6 + LLM judge at 0.4) achieves a **Pearson correlation of r = 0.98** with human ratings of overall sales performance, which is the result that justifies using the automated score in place of human annotation at benchmark scale. |
|
|
| Accuracy is reported over the full 5-class problem. Note that the classes are not balanced in realistic sales data — successful closes are rarer than non-purchases, and `F` (abusive) is rarest of all — so per-class recall on the tail classes will be lower than the aggregate figure suggests. If your use case hinges on detecting `F` or `X` specifically, measure per-class performance on your own data before relying on it. |
|
|
| --- |
|
|
| ## Limitations and Risks |
|
|
| - **Truncation is the main failure mode.** Feed it head-truncated text and predictions degrade badly while still looking confident. Always tail-truncate. |
| - **128-token window.** Intent expressed early and never restated near the end will be missed. Conversations that end with an off-topic exchange can also mislead it. |
| - **Trained on simulated + real sales dialogue** in Financial Services and Consumer Goods. Other verticals, other conversation formats (email threads, support tickets), and non-sales dialogue are out of distribution. |
| - **Format-coupled.** The `[ASSISTANT]`/`[USER]` tagging is part of the learned input representation, not a cosmetic choice. Different tags or added whitespace will shift predictions. |
| - **Not a purchase predictor.** It classifies *expressed* intent at the end of a conversation. Stated intent is not a real-world conversion rate, and it should not be used to forecast revenue or to score individual human salespeople for performance management. |
| - **Ordinal scores are a benchmark convention.** The 10/8/6/4/2 mapping was chosen for the SalesLLM metric. The intervals are not calibrated probabilities and should not be treated as such. |
| - **Inherited bias.** Predictions may vary with dialect, phrasing formality, and translationese in ways that correlate with demographics. Do not use it to gate access, rank customers, or make decisions affecting individuals. |
|
|
| --- |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{salesllm, |
| title = {SalesLLM: Benchmarking LLM Realistic Selling Skill}, |
| author = {MultiSense}, |
| year = {2025}, |
| url = {https://github.com/Bairong-Xdynamics/Benchmarking-LLM-Realistic-Selling-Skill} |
| } |
| ``` |
|
|
| ## Related |
|
|
| - 📊 [SalesLLM benchmark & code](https://github.com/Bairong-Xdynamics/Benchmarking-LLM-Realistic-Selling-Skill) |
| - 🤗 [CustomerLM](https://huggingface.co/MultiSense/CustomerLM) — the user simulator that generates the customer side of the dialogues this model scores |
