v2.0: Semantic rule improvements + dataset expansion (+9082 samples)

README.md

@@ -0,0 +1,392 @@
+---
+language: tr
+license: other
+license_name: siriusai-premium-v1
+license_link: LICENSE
+tags:
+- turkish
+- text-classification
+- bert
+- nlp
+- transformers
+- turn-detection
+- voice-assistant
+- latency-optimization
+- siriusai
+- production-ready
+- enterprise
+base_model: dbmdz/bert-base-turkish-uncased
+datasets:
+- custom
+metrics:
+- f1
+- precision
+- recall
+- accuracy
+- mcc
+library_name: transformers
+pipeline_tag: text-classification
+model-index:
+- name: turn-detector-v2
+  results:
+  - task:
+      type: text-classification
+      name: Text Classification
+    metrics:
+    - type: f1
+      value: 0.9769
+      name: Macro F1
+    - type: mcc
+      value: 0.9544
+      name: MCC
+    - type: accuracy
+      value: 97.94
+      name: Accuracy
+---
+
+# turn-detector-v2 - Turkish Turn Detection Model
+
+<p align="center">
+  <a href="https://huggingface.co/hayatiali/turn-detector-v2"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-turn--detector--v2-yellow" alt="Hugging Face"></a>
+  <a href="https://huggingface.co/hayatiali/turn-detector-v2"><img src="https://img.shields.io/badge/Model-Production%20Ready-brightgreen" alt="Production Ready"></a>
+  <img src="https://img.shields.io/badge/Language-Turkish-blue" alt="Turkish">
+  <img src="https://img.shields.io/badge/Task-Turn%20Detection-orange" alt="Turn Detection">
+  <img src="https://img.shields.io/badge/F1-97.69%25-success" alt="F1 Score">
+</p>
+
+This model is designed for detecting turn-taking patterns in Turkish conversations, optimizing voice assistant latency by identifying when user utterances require LLM processing vs. simple acknowledgments.
+
+*Developed by SiriusAI Tech Brain Team*
+
+---
+
+## Mission
+
+> **To optimize voice assistant response latency by detecting when user utterances require LLM processing vs. simple acknowledgments.**
+
+The `turn-detector-v2` model analyzes **conversational turn pairs** (bot utterance + user response) and classifies whether the user's response requires LLM processing (**agent_response**) or is just a backchannel acknowledgment that can be handled without LLM (**backchannel**).
+
+### Key Benefits
+
+| Benefit | Description |
+|---------|-------------|
+| **Latency Reduction** | Skip LLM calls for backchannels, saving 500-2000ms per interaction |
+| **Cost Optimization** | Reduce LLM API costs by filtering unnecessary calls |
+| **Natural Conversation** | Return immediate filler responses ("hmm", "tamam") for acknowledgments |
+| **High Accuracy** | 97.94% accuracy ensures reliable real-world performance |
+
+---
+
+## Model Overview
+
+| Property | Value |
+|----------|-------|
+| **Architecture** | BertForSequenceClassification |
+| **Base Model** | `dbmdz/bert-base-turkish-uncased` |
+| **Task** | Binary Text Classification |
+| **Language** | Turkish (tr) |
+| **Labels** | 2 (agent_response, backchannel) |
+| **Model Size** | ~110M parameters |
+| **Inference Time** | ~10-15ms (GPU) / ~40-50ms (CPU) |
+
+---
+
+## Performance Metrics
+
+### Final Evaluation Results
+
+| Metric | Score |
+|--------|-------|
+| **Macro F1** | **0.9769** |
+| **Micro F1** | **0.9794** |
+| **MCC** | **0.9544** |
+| **Accuracy** | **97.94%** |
+
+### Per-Class Performance
+
+| Category | Accuracy | Samples |
+|----------|----------|---------|
+| **agent_response** | 99.57% | 8,553 |
+| **backchannel** | 94.83% | 4,470 |
+
+---
+
+## Semantic Classification Rules
+
+### When to Classify as `backchannel` (Skip LLM)
+
+| Condition | Examples |
+|-----------|----------|
+| Bot gives info + User short acknowledgment | "tamam", "anladim", "ok", "peki" |
+| Bot gives info + User rhetorical question | "oyle mi?", "harbi mi?", "cidden mi?" |
+| Bot gives info + User minimal response | "hmm", "hi hi", "evet" |
+
+### When to Classify as `agent_response` (Send to LLM)
+
+| Condition | Examples |
+|-----------|----------|
+| Bot asks question + User gives any answer | "[bot] adi nedir [sep] [user] ahmet" |
+| Bot gives info + User asks real question | "[bot] faturaniz kesildi [sep] [user] ne zaman?" |
+| Bot gives info + User makes request | "[bot] kargonuz yolda [sep] [user] adresi degistirmek istiyorum" |
+| User provides detailed information | "[bot] bilgi verir misiniz [sep] [user] sunu sunu istiyorum cunku..." |
+
+### Golden Rule
+
+```
+If bot asked a question → Always agent_response
+If bot gave info + User short acknowledgment → backchannel
+```
+
+---
+
+## Dataset
+
+### Dataset Statistics
+
+| Split | Samples |
+|-------|---------|
+| **Train** | 52,287 |
+| **Test** | 13,023 |
+| **Total** | 65,310 |
+
+### Label Distribution
+
+| Label | Count | Percentage |
+|-------|-------|------------|
+| **agent_response** | 35,223 | 67.4% |
+| **backchannel** | 17,064 | 32.6% |
+
+### Domain Coverage
+
+- E-commerce (kargo, iade, teslimat)
+- Banking (hesap, bakiye, kredi)
+- Telecom (numara tasima, data, hat)
+- Insurance (prim, police, teminat, kasko)
+- General Support (sikayet, yonetici, eskalasyon)
+- Identity Verification (TC, gorusuyorum, soyadi)
+
+---
+
+## Label Definitions
+
+| Label | ID | Description |
+|-------|-----|-------------|
+| **agent_response** | 0 | User response requires LLM processing - questions, requests, confirmations to questions, corrections |
+| **backchannel** | 1 | Simple acknowledgment - LLM skipped, filler returned (tamam, anladim, ok) |
+
+### Input Format
+
+```
+[bot] <bot utterance> [sep] [user] <user response>
+```
+
+### Example Classifications
+
+**agent_response** (Send to LLM):
+```
+[bot] size nasil yardimci olabilirim [sep] [user] fatura sorgulamak istiyorum
+[bot] ahmet bey ile mi gorusuyorum [sep] [user] evet benim
+[bot] islemi onayliyor musunuz [sep] [user] evet onayliyorum
+[bot] kargonuz yolda [sep] [user] ne zaman gelir
+[bot] poliçeniz aktif [sep] [user] teminat limitini ogrenebilir miyim
+```
+
+**backchannel** (Skip LLM, return filler):
+```
+[bot] faturaniz 150 tl gorunuyor [sep] [user] tamam
+[bot] siparisiniz 3 gun icinde teslim edilecek [sep] [user] anladim
+[bot] kaydinizi kontrol ediyorum [sep] [user] peki
+[bot] policeniz yenilendi [sep] [user] tesekkurler
+[bot] sifreni sms ile gonderdik [sep] [user] ok aldim
+```
+
+---
+
+## Training
+
+### Hyperparameters
+
+| Parameter | Value |
+|-----------|-------|
+| **Base Model** | `dbmdz/bert-base-turkish-uncased` |
+| **Max Sequence Length** | 128 tokens |
+| **Batch Size** | 16 |
+| **Learning Rate** | 3e-5 |
+| **Epochs** | 4 |
+| **Optimizer** | AdamW |
+| **Weight Decay** | 0.01 |
+| **Loss Function** | CrossEntropyLoss |
+| **Hardware** | Apple Silicon (MPS) |
+
+---
+
+## Usage
+
+### Installation
+
+```bash
+pip install transformers torch
+```
+
+### Quick Start
+
+```python
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+import torch
+
+model_name = "hayatiali/turn-detector-v2"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForSequenceClassification.from_pretrained(model_name)
+model.eval()
+
+LABELS = ["agent_response", "backchannel"]
+
+def predict(text):
+    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
+    with torch.no_grad():
+        outputs = model(**inputs)
+        probs = torch.softmax(outputs.logits, dim=-1)[0]
+
+    scores = {label: float(prob) for label, prob in zip(LABELS, probs)}
+    return {"label": max(scores, key=scores.get), "confidence": max(scores.values())}
+
+# Bot asks question → agent_response
+print(predict("[bot] ahmet bey ile mi gorusuyorum [sep] [user] evet benim"))
+# Output: {'label': 'agent_response', 'confidence': 0.99}
+
+# Bot gives info + User acknowledges → backchannel
+print(predict("[bot] faturaniz 150 tl gorunuyor [sep] [user] tamam"))
+# Output: {'label': 'backchannel', 'confidence': 0.98}
+```
+
+### Production Integration
+
+```python
+class TurnDetector:
+    """Production-ready turn detection for voice assistants."""
+
+    LABELS = ["agent_response", "backchannel"]
+    FILLER_RESPONSES = ["hmm", "evet", "tamam", "anlıyorum"]
+
+    def __init__(self, model_path="hayatiali/turn-detector-v2"):
+        self.tokenizer = AutoTokenizer.from_pretrained(model_path)
+        self.model = AutoModelForSequenceClassification.from_pretrained(model_path)
+        self.device = "cuda" if torch.cuda.is_available() else "cpu"
+        self.model.to(self.device).eval()
+
+    def should_call_llm(self, bot_text: str, user_text: str) -> dict:
+        """
+        Determines if user response should go to LLM.
+
+        Returns:
+            dict with 'call_llm' (bool), 'label', 'confidence', 'filler' (if backchannel)
+        """
+        text = f"[bot] {bot_text} [sep] [user] {user_text}"
+        inputs = self.tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
+        inputs = {k: v.to(self.device) for k, v in inputs.items()}
+
+        with torch.no_grad():
+            probs = torch.softmax(self.model(**inputs).logits, dim=-1)[0].cpu()
+
+        label_idx = probs.argmax().item()
+        label = self.LABELS[label_idx]
+        confidence = probs[label_idx].item()
+
+        result = {
+            "call_llm": label == "agent_response",
+            "label": label,
+            "confidence": confidence
+        }
+
+        if label == "backchannel":
+            import random
+            result["filler"] = random.choice(self.FILLER_RESPONSES)
+
+        return result
+
+# Usage
+detector = TurnDetector()
+
+# Case 1: Bot asks, user confirms → Send to LLM
+result = detector.should_call_llm("siparis iptal etmek ister misiniz", "evet iptal et")
+# {'call_llm': True, 'label': 'agent_response', 'confidence': 0.99}
+
+# Case 2: Bot informs, user acknowledges → Return filler
+result = detector.should_call_llm("siparisiz yola cikti", "tamam")
+# {'call_llm': False, 'label': 'backchannel', 'confidence': 0.97, 'filler': 'hmm'}
+```
+
+---
+
+## Limitations
+
+| Limitation | Details |
+|------------|---------|
+| **Language** | Turkish only, may struggle with heavy dialects |
+| **Context** | Single-turn analysis, no multi-turn memory |
+| **Domain** | Trained on customer service, may need fine-tuning for other domains |
+| **Edge Cases** | Ambiguous short responses may have lower confidence |
+
+---
+
+## Citation
+
+```bibtex
+@misc{turn-detector-v2-2025,
+  title={turn-detector-v2: Turkish Turn Detection for Voice Assistants},
+  author={SiriusAI Tech Brain Team},
+  year={2025},
+  publisher={Hugging Face},
+  howpublished={\url{https://huggingface.co/hayatiali/turn-detector-v2}},
+  note={Fine-tuned from dbmdz/bert-base-turkish-uncased}
+}
+```
+
+---
+
+## Contact
+
+- **Developer**: SiriusAI Tech Brain Team
+- **Email**: info@siriusaitech.com
+- **Repository**: [GitHub](https://github.com/sirius-tedarik)
+
+---
+
+## Changelog
+
+### v2.0 (Current)
+
+**Semantic Rule Improvements:**
+- If bot asks a question → always `agent_response` (731 rows corrected)
+- Rhetorical questions ("really?", "is that so?") → remain as `backchannel`
+- If user asks a real question ("when?", "how?") → `agent_response`
+
+**Dataset Expansion (+9,082 samples):**
+
+| Category | Added Patterns |
+|----------|----------------|
+| **Insurance** | premium, policy, coverage, comprehensive, interest, late fees |
+| **Telecom** | number porting, data exhausted, line transfer, GB remaining |
+| **E-commerce** | shipping cost, free shipping, returns, delivery |
+| **Price/Budget** | expensive, budget, too much, will think about it, not suitable |
+| **Identity Verification** | national ID, "am I speaking with...", surname, date of birth |
+| **Objection/Complaint** | unacceptable, not satisfied, complaint, impossible |
+| **Escalation** | manager, director, supervisor |
+| **Hold Requests** | one moment, busy right now, not now, later |
+
+**Metrics:** Macro F1: 0.9769, Accuracy: 97.94%
+
+> Note: Metrics appear slightly lower than v1.0, but this is a more accurate model.
+> v1.0 had mislabeled data (bot asked question + "yes" = backchannel),
+> which the model memorized. v2.0 ensures semantic consistency.
+
+### v1.0
+- Initial release
+- Dataset: 56,228 samples
+- Macro F1: 0.9924, Accuracy: 99.3%
+
+---
+
+**License**: SiriusAI Tech Premium License v1.0
+
+**Commercial Use**: Requires Premium License. Contact: info@siriusaitech.com

config.json

@@ -0,0 +1,119 @@
+{
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "agent_response",
+    "1": "backchannel"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "agent_response": 0,
+    "backchannel": 1
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "problem_type": "single_label_classification",
+  "torch_dtype": "float32",
+  "transformers_version": "4.52.4",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 32000,
+  "_metadata": {
+    "model_name": "turn-detection-v2",
+    "version": "1.0.0",
+    "published_at": "2025-12-31",
+    "author": "Fine-Tune Assistant",
+    "license": "Apache-2.0",
+    "huggingface_repo": "hayatiali/turn-detection-v2",
+    "huggingface_url": "https://huggingface.co/hayatiali/turn-detection-v2"
+  },
+  "_context_aware": {
+    "enabled": true,
+    "input_format": "[bot] {bot_message} [sep] [user] {user_message}",
+    "special_tokens": [
+      "[bot]",
+      "[sep]",
+      "[user]"
+    ],
+    "example_input": "[bot] sunucuya katilmak icin ne yapmaliyim [sep] [user] ya davet kodu alabilir miyim",
+    "fallback_behavior": "If no [bot] context provided, model uses user text only"
+  },
+  "_task": {
+    "type": "text-classification",
+    "name": "Turn Detection V2",
+    "description": "Classifies text into 2 categories: agent_response, backchannel",
+    "num_labels": 2
+  },
+  "_labels": {
+    "num_labels": 2,
+    "id2label": {
+      "0": "agent_response",
+      "1": "backchannel"
+    },
+    "label2id": {
+      "agent_response": 0,
+      "backchannel": 1
+    },
+    "label_descriptions": {
+      "agent_response": "Category: agent_response",
+      "backchannel": "Category: backchannel"
+    }
+  },
+  "_domain": {
+    "language": "Turkish (tr)",
+    "domain": "General",
+    "base_model": "dbmdz/bert-base-turkish-uncased"
+  },
+  "_training": {
+    "dataset": {
+      "name": "callcenter-turn-detection-classification",
+      "total_samples": 65310,
+      "train_samples": 52287,
+      "test_samples": 13023,
+      "label_distribution": {
+        "agent_response": "35223 (67.4%)",
+        "backchannel": "17064 (32.6%)"
+      }
+    },
+    "hyperparameters": {
+      "max_sequence_length": 128,
+      "batch_size": 16,
+      "learning_rate": 3e-05,
+      "epochs": 4,
+      "optimizer": "AdamW",
+      "weight_decay": 0.01,
+      "loss_function": "CrossEntropyLoss"
+    },
+    "hardware": "mps"
+  },
+  "_evaluation": {
+    "metrics": {
+      "macro_f1": 0.9769,
+      "micro_f1": 0.9794,
+      "mcc": 0.9544,
+      "accuracy": 97.94
+    },
+    "per_class": {
+      "agent_response": {
+        "accuracy": 99.57,
+        "samples": 8553
+      },
+      "backchannel": {
+        "accuracy": 94.83,
+        "samples": 4470
+      }
+    }
+  }
+}

evaluation_results.json

@@ -0,0 +1,25 @@
+{
+  "overall": {
+    "macro_f1": 0.9769330455276665,
+    "micro_f1": 0.9794210243415495,
+    "mcc": 0.9544096525818544,
+    "accuracy": 97.94210243415495
+  },
+  "per_class": {
+    "agent_response": {
+      "accuracy": 99.56740325032153,
+      "correct": 8516,
+      "total": 8553
+    },
+    "backchannel": {
+      "accuracy": 94.83221476510067,
+      "correct": 4239,
+      "total": 4470
+    }
+  },
+  "labels": [
+    "agent_response",
+    "backchannel"
+  ],
+  "evaluated_at": "2025-12-31T22:16:17.601187"
+}

label_config.json

@@ -0,0 +1,17 @@
+{
+  "labels": [
+    "agent_response",
+    "backchannel"
+  ],
+  "id2label": {
+    "0": "agent_response",
+    "1": "backchannel"
+  },
+  "label2id": {
+    "agent_response": 0,
+    "backchannel": 1
+  },
+  "num_labels": 2,
+  "base_model": "dbmdz/bert-base-turkish-uncased",
+  "trained_at": "2025-12-31T22:15:45.605311"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27154b013aebd7565d1d90b6de65fe54e24c66823b1c985605ff9cbda40bcd89
+size 442499064

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,59 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "max_len": 512,
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ce37056b653bec41c689d47ad2e5467eee5c03e8e20cf17c2a79dd05dfaa8f1
+size 5841