Update README.md

README.md

@@ -1,3 +1,186 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-classification
+language:
+- zh
+- en
+tags:
+- bert
+- text-classification
+- sales
+- intent-classification
+- dialogue
+- evaluation
+---
+
+# SaleIntent-BERT
+
+**SaleIntent-BERT** is a fine-tuned BERT classifier that reads a complete sales conversation and predicts **how the customer ended up** — from clear purchase intent down to hostility. It is the outcome-scoring half of the [SalesLLM benchmark](https://github.com/Bairong-Xdynamics/Benchmarking-LLM-Realistic-Selling-Skill).
+
+Scoring a sales dialogue with an LLM judge alone conflates two different questions: *did the salesperson run a good process?* and *did the customer actually want to buy at the end?* A model can be articulate, polite, and well-structured while the customer walks away — and an LLM judge, reading the whole transcript, tends to reward the articulate process. SaleIntent-BERT answers the second question independently, by looking only at where the conversation landed.
+
+It reaches **93.51% accuracy on Chinese and 92.94% on English**, and pairs with the LLM judge to produce the final SalesLLM Score.
+
+| | |
+| :--- | :--- |
+| **Task** | 5-class sequence classification over a full dialogue |
+| **Base model** | BERT (see `config.json` for the exact checkpoint) |
+| **Languages** | Chinese, English |
+| **Input** | Flattened multi-turn dialogue, **last 128 tokens** |
+| **Accuracy** | 93.51% (ZH), 92.94% (EN) |
+| **License** | Apache 2.0 |
+
+---
+
+## Labels
+
+The five classes are an **outcome grade**, not a monotonic intent ladder. Each maps to a point score used in the final benchmark metric:
+
+| Label | Meaning | Score |
+| :---: | :--- | :---: |
+| **A** | Customer has **clear** purchase intent | 10 |
+| **B** | Customer **possibly** has intent | 8 |
+| **C** | Customer has **no** purchase intent | 6 |
+| **X** | Customer has **weak** intent; dismissive / going through the motions | 4 |
+| **F** | Customer is **abusive or complaining** | 2 |
+
+Read the ordering carefully — **C (no intent) scores higher than X (weak, dismissive)**. A clean, honest "no" is a better conversational outcome than one the salesperson dragged into disengaged stonewalling, and an outright hostile ending (F) is worst of all. The scale grades the *state the salesperson left the customer in*, not just how close the sale was.
+
+Class indices are `A=0, B=1, C=2, F=3, X=4`. Do not assume index order matches score order — always resolve through `model.config.id2label`.
+
+---
+
+## Input Format
+
+The model expects the dialogue **flattened into a single string** with explicit speaker tags, then **tail-truncated to the last 128 tokens**:
+
+```python
+def flatten_dialogue(messages):
+    out = ""
+    for msg in messages:
+        tag = "[ASSISTANT]" if msg["role"] == "assistant" else "[USER]"
+        out += tag + msg["content"]
+    return out
+```
+
+Two details are load-bearing:
+
+- **Speaker tags, no separators.** `[ASSISTANT]`/`[USER]` are concatenated directly against the message text with no spaces or newlines. The model was trained on exactly this string shape.
+- **Tail truncation, not head.** Buying intent is decided at the *end* of a conversation, so the last 128 tokens are kept and everything before is dropped. Standard `truncation=True` keeps the *head* and will silently feed the model the opening pleasantries instead of the outcome — this is the single most common way to get bad predictions from this model.
+
+`last_token_num=128` is the benchmark's validated setting. The window is deliberately short: a longer window pulls in mid-conversation negotiation that dilutes the end-state signal.
+
+---
+
+## Usage
+
+### Direct inference
+
+```python
+import torch
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+model_id = "MultiSense/SaleIntent_bert"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()
+
+LABEL2SCORE = {"A": 10, "B": 8, "C": 6, "X": 4, "F": 2}
+N, MAX_LEN = 128, 512
+
+def flatten_dialogue(messages):
+    return "".join(
+        ("[ASSISTANT]" if m["role"] == "assistant" else "[USER]") + m["content"]
+        for m in messages
+    )
+
+def encode_tail(text, n=N, max_length=MAX_LEN):
+    """Keep the LAST n tokens — intent lives at the end of the dialogue."""
+    toks = tok.tokenize(text)[-min(n, max_length - 2):]
+    ids = [tok.cls_token_id] + tok.convert_tokens_to_ids(toks) + [tok.sep_token_id]
+    mask = [1] * len(ids)
+    pad = max_length - len(ids)
+    return {
+        "input_ids": torch.tensor([ids + [tok.pad_token_id] * pad]),
+        "attention_mask": torch.tensor([mask + [0] * pad]),
+    }
+
+messages = [
+    {"role": "user",      "content": "你好，我想了解一下你们的降噪耳机。"},
+    {"role": "assistant", "content": "好的，这款支持32dB混合降噪，续航38小时，售价1999元。"},
+    {"role": "user",      "content": "听起来不错，那我下单一个吧。"},
+]
+
+with torch.no_grad():
+    logits = model(**encode_tail(flatten_dialogue(messages))).logits
+
+label = model.config.id2label[logits.argmax(-1).item()]
+print(label, LABEL2SCORE[label])   # -> A 10
+```
+
+### As part of the SalesLLM score
+
+`comprehensive_score.py` runs this model over a results file and blends it with the LLM judge:
+
+```bash
+python salesllm/comprehensive_score.py \
+  --bert_path "MultiSense/SaleIntent_bert" \
+  --source_file "./results/zh/<output>.jsonl" \
+  --llm_model_name "<judge_model>" \
+  --api_key "<key>" --end_point "<base_url>" \
+  --last_token_num 128 \
+  --proportion 0.6
+```
+
+The final score is a weighted blend of the two signals:
+
+```
+final_score = proportion * LABEL2SCORE[bert_label] + (1 - proportion) * llm_judge_score
+```
+
+The benchmark uses **`proportion = 0.6`** — outcome weighted slightly above process, because outcome is the harder signal to game. Both components are on the same 0–10 scale, so the blend is directly interpretable.
+
+Output is written to `<source_file>_scored.json`, with `A/B` (the predicted label), `conversation_quality` (the LLM judge's 0–10), and `final_score` added to each record.
+
+---
+
+## Evaluation
+
+| Language | Accuracy |
+| :--- | :---: |
+| Chinese | **93.51%** |
+| English | **92.94%** |
+
+The combined pipeline (this classifier at 0.6 + LLM judge at 0.4) achieves a **Pearson correlation of r = 0.98** with human ratings of overall sales performance, which is the result that justifies using the automated score in place of human annotation at benchmark scale.
+
+Accuracy is reported over the full 5-class problem. Note that the classes are not balanced in realistic sales data — successful closes are rarer than non-purchases, and `F` (abusive) is rarest of all — so per-class recall on the tail classes will be lower than the aggregate figure suggests. If your use case hinges on detecting `F` or `X` specifically, measure per-class performance on your own data before relying on it.
+
+---
+
+## Limitations and Risks
+
+- **Truncation is the main failure mode.** Feed it head-truncated text and predictions degrade badly while still looking confident. Always tail-truncate.
+- **128-token window.** Intent expressed early and never restated near the end will be missed. Conversations that end with an off-topic exchange can also mislead it.
+- **Trained on simulated + real sales dialogue** in Financial Services and Consumer Goods. Other verticals, other conversation formats (email threads, support tickets), and non-sales dialogue are out of distribution.
+- **Format-coupled.** The `[ASSISTANT]`/`[USER]` tagging is part of the learned input representation, not a cosmetic choice. Different tags or added whitespace will shift predictions.
+- **Not a purchase predictor.** It classifies *expressed* intent at the end of a conversation. Stated intent is not a real-world conversion rate, and it should not be used to forecast revenue or to score individual human salespeople for performance management.
+- **Ordinal scores are a benchmark convention.** The 10/8/6/4/2 mapping was chosen for the SalesLLM metric. The intervals are not calibrated probabilities and should not be treated as such.
+- **Inherited bias.** Predictions may vary with dialect, phrasing formality, and translationese in ways that correlate with demographics. Do not use it to gate access, rank customers, or make decisions affecting individuals.
+
+---
+
+## Citation
+
+```bibtex
+@misc{salesllm,
+  title  = {SalesLLM: Benchmarking LLM Realistic Selling Skill},
+  author = {MultiSense},
+  year   = {2025},
+  url    = {https://github.com/Bairong-Xdynamics/Benchmarking-LLM-Realistic-Selling-Skill}
+}
+```
+
+## Related
+
+- 📊 [SalesLLM benchmark & code](https://github.com/Bairong-Xdynamics/Benchmarking-LLM-Realistic-Selling-Skill)
+- 🤗 [CustomerLM](https://huggingface.co/MultiSense/CustomerLM) — the user simulator that generates the customer side of the dialogues this model scores