Update README.md
Browse files
README.md
CHANGED
|
@@ -114,7 +114,7 @@ datasets:
|
|
| 114 |
- ZombitX64/Sentiment-Benchmark
|
| 115 |
---
|
| 116 |
|
| 117 |
-
# MultiSent-E5
|
| 118 |
|
| 119 |
<div align="center">
|
| 120 |
<picture>
|
|
@@ -221,7 +221,7 @@ from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
|
| 221 |
import torch
|
| 222 |
|
| 223 |
# Load the model and tokenizer
|
| 224 |
-
model_name = "ZombitX64/MultiSent-E5"
|
| 225 |
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
| 226 |
model = AutoModelForSequenceClassification.from_pretrained(model_name)
|
| 227 |
|
|
@@ -280,8 +280,8 @@ from transformers import pipeline
|
|
| 280 |
|
| 281 |
# Create a sentiment analysis pipeline
|
| 282 |
classifier = pipeline("text-classification",
|
| 283 |
-
model="ZombitX64/MultiSent-E5",
|
| 284 |
-
tokenizer="ZombitX64/MultiSent-E5")
|
| 285 |
|
| 286 |
# Analyze sentiment
|
| 287 |
texts = [
|
|
@@ -364,148 +364,56 @@ The model showed excellent convergence with minimal overfitting:
|
|
| 364 |
- Early convergence suggests effective transfer learning from the base model
|
| 365 |
|
| 366 |
============================================================
|
| 367 |
-
Evaluating
|
| 368 |
============================================================
|
| 369 |
-
|
| 370 |
-
|
| 371 |
-
|
| 372 |
-
|
| 373 |
-
|
| 374 |
-
|
| 375 |
-
|
| 376 |
-
|
| 377 |
-
|
| 378 |
-
|
| 379 |
-
|
| 380 |
-
|
| 381 |
-
|
| 382 |
-
|
| 383 |
-
|
| 384 |
-
|
| 385 |
-
neutral -> question 1
|
| 386 |
-
Name: count, dtype: int64
|
| 387 |
-
|
| 388 |
-
Low Confidence Errors (< 60%): 4
|
| 389 |
-
High Confidence Errors (> 80%): 19
|
| 390 |
-
|
| 391 |
-
=== ERROR EXAMPLES ===
|
| 392 |
-
|
| 393 |
-
negative -> positive:
|
| 394 |
-
Text: 'สุดยอดไปเลย! เธอใช้เวลาทั้งวันทำงานชิ้นนี้ออกมาได้แค่นี้เองเหรอเนี่ย!'
|
| 395 |
-
Confidence: 0.517
|
| 396 |
-
Text: 'ไอเดียสร้างสรรค์มาก! ไม่มีใครคิดจะเสนออะไรที่ไม่มีทางเป็นไปได้แบบนี้หรอก'
|
| 397 |
-
Confidence: 1.000
|
| 398 |
-
Text: 'ไอเดียสร้างสรรค์มาก! ไม่มีใครคิดจะเสนออะไรที่ไม่มีทางเป็นไปได้แบบนี้หรอก'
|
| 399 |
-
Confidence: 1.000
|
| 400 |
-
|
| 401 |
-
question -> neutral:
|
| 402 |
-
Text: 'คุณคิดว่าอย่างไรกับเรื่องนี้'
|
| 403 |
-
Confidence: 0.999
|
| 404 |
-
Text: 'How was your day today?'
|
| 405 |
-
Confidence: 1.000
|
| 406 |
-
Text: '你觉得怎么样?'
|
| 407 |
-
Confidence: 0.999
|
| 408 |
-
|
| 409 |
-
neutral -> positive:
|
| 410 |
-
Text: 'ก็แข็งแรงอยู่นะ'
|
| 411 |
-
Confidence: 0.727
|
| 412 |
-
Text: 'ก็แข็งแรงอยู่นะ'
|
| 413 |
-
Confidence: 0.727
|
| 414 |
-
Text: 'บรรยากาศดีมาก เหมาะกับการนั่งเงียบๆ คนเดียว'
|
| 415 |
-
Confidence: 0.723
|
| 416 |
-
|
| 417 |
-
negative -> neutral:
|
| 418 |
-
Text: 'Good day. Unfortunately, I had to walk 10 kilometers from home to school, and now I'm feeling quite ...'
|
| 419 |
-
Confidence: 0.970
|
| 420 |
-
Text: 'Good day. Unfortunately, I had to walk 10 kilometers from home to school, and now I'm feeling quite ...'
|
| 421 |
-
Confidence: 0.970
|
| 422 |
-
Text: 'ส่งของไวมาก...ถ้านับวันเป็นเดือน'
|
| 423 |
-
Confidence: 0.999
|
| 424 |
-
|
| 425 |
-
question -> positive:
|
| 426 |
-
Text: 'ลำไยอร่อยดีสดมากและลูกใหญ่ด้วยแต่เน่าไปครึ่งนึงมั้ย'
|
| 427 |
-
Confidence: 0.550
|
| 428 |
-
Text: 'ลำไยอร่อยดีสดมากและลูกใหญ่ด้วยแต่เน่าไปครึ่งนึงมั้ย'
|
| 429 |
-
Confidence: 0.550
|
| 430 |
-
|
| 431 |
-
=== LOW CONFIDENCE PREDICTIONS ===
|
| 432 |
-
Total Low Confidence: 7 (4.1%)
|
| 433 |
-
|
| 434 |
-
Low Confidence Examples:
|
| 435 |
-
'ลำไยอร่อยดีสดมากและลูกใหญ่ด้วยแต่เน่าไปครึ่งนึงมั้ย'
|
| 436 |
-
Predicted: positive, Confidence: 0.550
|
| 437 |
-
True: question, Correct: False
|
| 438 |
-
|
| 439 |
-
'ลำไยอร่อยดีสดมากและลูกใหญ่ด้วยแต่เน่าไปครึ่งนึงรึเปล่า'
|
| 440 |
-
Predicted: question, Confidence: 0.521
|
| 441 |
-
True: question, Correct: True
|
| 442 |
-
|
| 443 |
-
'สุดยอดไปเลย! เธอใช้เวลาทั้งวันทำงานชิ้นนี้ออกมาได้แค่นี้เองเหรอเนี่ย!'
|
| 444 |
-
Predicted: positive, Confidence: 0.517
|
| 445 |
-
True: negative, Correct: False
|
| 446 |
-
|
| 447 |
-
'เกือบดีแล้วล่ะ เหลือแค่ดีจริงๆ นิดเดียว'
|
| 448 |
-
Predicted: neutral, Confidence: 0.546
|
| 449 |
-
True: neutral, Correct: True
|
| 450 |
-
|
| 451 |
-
'ลำไยอร่อยดีสดมากและลูกใหญ่ด้วยแต่เน่าไปคร��่งนึงมั้ย'
|
| 452 |
-
Predicted: positive, Confidence: 0.550
|
| 453 |
-
True: question, Correct: False
|
| 454 |
-
|
| 455 |
-
|
| 456 |
-
### 📊 **สรุปผลการประเมินโมเดล: MultiSent-E5**
|
| 457 |
-
|
| 458 |
-
| Metric | ค่า (Value) |
|
| 459 |
-
| --------------------------------- | ----------- |
|
| 460 |
-
| **Accuracy** | **84.9%** |
|
| 461 |
-
| **F1 Macro** | 83.9% |
|
| 462 |
-
| **F1 Weighted** | 85.0% |
|
| 463 |
-
| **จำนวนตัวอย่างทั้งหมด** | 172 |
|
| 464 |
-
| **จำนวนข้อผิดพลาด (Error)** | 26 |
|
| 465 |
-
| **เปอร์เซ็นต์ความผิดพลาด** | 15.1% |
|
| 466 |
-
| **Low Confidence Errors (<60%)** | 4 |
|
| 467 |
-
| **High Confidence Errors (>80%)** | 19 |
|
| 468 |
|
| 469 |
-
|
| 470 |
-
|
| 471 |
-
|
| 472 |
-
|
| 473 |
-
| ผิดจาก (True Label) | เป็น (Predicted Label) | จำนวนครั้ง (Count) |
|
| 474 |
-
| ------------------- | ---------------------- | ------------------ |
|
| 475 |
-
| negative | positive | 7 |
|
| 476 |
-
| question | neutral | 7 |
|
| 477 |
-
| neutral | positive | 3 |
|
| 478 |
-
| negative | neutral | 3 |
|
| 479 |
-
| question | positive | 2 |
|
| 480 |
-
| positive | neutral | 2 |
|
| 481 |
-
| neutral | negative | 1 |
|
| 482 |
-
| neutral | question | 1 |
|
| 483 |
-
|
| 484 |
-
---
|
| 485 |
-
|
| 486 |
-
### 🔍 **ตัวอย่าง Error ที่น่าสนใจ**
|
| 487 |
-
|
| 488 |
-
#### 1. **ประชด/เสียดสี** ที่ผิดเป็น Positive
|
| 489 |
-
|
| 490 |
-
| ข้อความ | ทำนาย | จริง | Confidence |
|
| 491 |
-
| ------------------------------------------------------------------------ | -------- | -------- | ---------- |
|
| 492 |
-
| สุดยอดไปเลย! เธอใช้เวลาทั้งวันทำงานชิ้นนี้ออกมาได้แค่นี้เองเหรอเนี่ย! | positive | negative | 0.517 |
|
| 493 |
-
| ไอเดียสร้างสรรค์มาก! ไม่มีใครคิดจะเสนออะไรที่ไม่มีทางเป็นไปได้แบบนี้หรอก | positive | negative | 1.000 |
|
| 494 |
-
|
| 495 |
-
#### 2. **คำถาม** ผิดเป็น Neutral
|
| 496 |
-
|
| 497 |
-
| ข้อความ | ทำนาย | จริง | Confidence |
|
| 498 |
-
| ---------------------------- | ------- | -------- | ---------- |
|
| 499 |
-
| คุณคิดว่าอย่างไรกับเรื่องนี้ | neutral | question | 0.999 |
|
| 500 |
-
| How was your day today? | neutral | question | 1.000 |
|
| 501 |
-
| 你觉得怎么样? | neutral | question | 0.999 |
|
| 502 |
-
|
| 503 |
-
#### 3. **ประโยคคลุมเครือที่ Low Confidence**
|
| 504 |
|
| 505 |
-
|
| 506 |
-
|
| 507 |
-
|
| 508 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 509 |
|
| 510 |
|
| 511 |
|
|
@@ -515,7 +423,7 @@ Low Confidence Examples:
|
|
| 515 |
|
| 516 |
The model was evaluated on a carefully selected validation set with the following characteristics:
|
| 517 |
|
| 518 |
-
* **Total
|
| 519 |
* **Selection Method:** Stratified random sampling to maintain class distribution
|
| 520 |
* **Data Quality:** Manually verified and cleaned validation samples
|
| 521 |
* **Evaluation Period:** Final model checkpoint from epoch 5
|
|
@@ -532,95 +440,6 @@ The model was comprehensively evaluated using multiple metrics:
|
|
| 532 |
- **Recall:** Per-class and overall recall scores
|
| 533 |
- **Support:** Number of samples per class in validation set
|
| 534 |
|
| 535 |
-
### Results
|
| 536 |
-
|
| 537 |
-
#### Final Test Results
|
| 538 |
-
|
| 539 |
-
**Per-Class Performance:**
|
| 540 |
-
|
| 541 |
-
| Class | Precision | Recall | F1-Score | Support | Performance Notes |
|
| 542 |
-
|-------|-----------|---------|----------|---------|-------------------|
|
| 543 |
-
| Question | N/A | N/A | N/A | 0 | No question samples in validation set |
|
| 544 |
-
| Negative | 1.00 | 1.00 | 1.00 | 231 | Perfect classification |
|
| 545 |
-
| Neutral | 1.00 | 0.90 | 0.95 | 10 | 1 misclassification due to small sample size |
|
| 546 |
-
| Positive | 1.00 | 1.00 | 1.00 | 32 | Perfect classification |
|
| 547 |
-
|
| 548 |
-
**Overall Performance Summary:**
|
| 549 |
-
|
| 550 |
-
| Metric | Value | Interpretation |
|
| 551 |
-
|--------|-------|----------------|
|
| 552 |
-
| **Overall Accuracy** | 100% (273/273) | Exceptional performance |
|
| 553 |
-
| **Macro Average F1** | 0.98 | Excellent across all represented classes |
|
| 554 |
-
| **Weighted Average F1** | 1.00 | Perfect when weighted by class frequency |
|
| 555 |
-
| **Total Correct Predictions** | 272/273 | Only 1 misclassification |
|
| 556 |
-
|
| 557 |
-
#### Detailed Confusion Matrix Results
|
| 558 |
-
|
| 559 |
-
**Classification Breakdown:**
|
| 560 |
-
- **Negative Class:** 231/231 correctly classified (100% accuracy)
|
| 561 |
-
- **Neutral Class:** 9/10 correctly classified (90% accuracy)
|
| 562 |
-
- 1 neutral sample misclassified (likely as positive due to ambiguous language)
|
| 563 |
-
- **Positive Class:** 32/32 correctly classified (100% accuracy)
|
| 564 |
-
- **Question Class:** Not present in validation set
|
| 565 |
-
|
| 566 |
-
### Model Capabilities
|
| 567 |
-
|
| 568 |
-
#### Demonstrated Strengths
|
| 569 |
-
|
| 570 |
-
The model shows exceptional capability in understanding various aspects of Thai sentiment:
|
| 571 |
-
|
| 572 |
-
**1. Straightforward Sentiment Classification:**
|
| 573 |
-
- Clear positive expressions: "วันนี้อากาศดีจังเลย" (The weather is so nice today) → Positive (99.96%)
|
| 574 |
-
- Clear negative expressions: "แย่ที่สุดเท่าที่เคยเจอมา" (The worst I've ever encountered) → Negative (99.99%)
|
| 575 |
-
- Neutral expressions: "ก็งั้นๆ แหละ ไม่มีอะไรพิเศษ" (It's just okay, nothing special) → Neutral (99.70%)
|
| 576 |
-
|
| 577 |
-
**2. Advanced Linguistic Understanding:**
|
| 578 |
-
|
| 579 |
-
**Sarcasm Detection:**
|
| 580 |
-
- "เก่งจังเลยนะ ทำผิดซ้ำได้เหมือนเดิมเป๊ะเลย"
|
| 581 |
-
(So talented! You can make the same mistake repeatedly) → Negative (99.99%)
|
| 582 |
-
- The model correctly identifies that "เก่งจัง" (so talented) is used sarcastically
|
| 583 |
-
|
| 584 |
-
**Implicit Criticism:**
|
| 585 |
-
- "ไอเดียสร้างสรรค์มาก! ไม่มีใครคิดจะเสนออะไรที่ไม่มีทางเป็นไปได้แบบนี้หรอก"
|
| 586 |
-
(Very creative idea! No one would think to propose something this impossible) → Negative (99.43%)
|
| 587 |
-
- Successfully detects negative sentiment despite seemingly positive words
|
| 588 |
-
|
| 589 |
-
**3. Cultural Context Understanding:**
|
| 590 |
-
- Thai-specific expressions and idioms
|
| 591 |
-
- Formal vs. informal language registers
|
| 592 |
-
- Regional variations in expression
|
| 593 |
-
|
| 594 |
-
#### Performance Analysis by Text Type
|
| 595 |
-
|
| 596 |
-
| Text Type | Accuracy | Confidence Range | Notes |
|
| 597 |
-
|-----------|----------|------------------|--------|
|
| 598 |
-
| Direct statements | 99-100% | 90-100% | Excellent performance |
|
| 599 |
-
| Sarcastic content | 95-99% | 85-99% | Very good sarcasm detection (e.g., "เก่งจังเลยนะ ทำผิดซ้ำได้เหมือนเดิมเป๊ะเลย" → 99.98% negative) |
|
| 600 |
-
| Implicit sentiment | 90-95% | 80-95% | Good at reading between the lines |
|
| 601 |
-
| **Mixed sentiment** | **60-75%** | **50-60%** | **Struggles with texts containing both positive and negative aspects** |
|
| 602 |
-
| **Question-like text** | **40-60%** | **50-60%** | **Poor question detection, often classified as other categories** |
|
| 603 |
-
| Star ratings | 95-100% | 99%+ | Excellent (e.g., "ให้5ดาวเลย" → 99.98% positive, "ให้1ดาวเลย" → 99.49% negative) |
|
| 604 |
-
| Formal language | 98-100% | 85-100% | Strong performance on formal text |
|
| 605 |
-
| Colloquial language | 95-99% | 80-95% | Handles informal text well |
|
| 606 |
-
|
| 607 |
-
#### Real-World Performance Issues
|
| 608 |
-
|
| 609 |
-
**Low Confidence Predictions (< 60%):**
|
| 610 |
-
Based on empirical testing, these text types frequently produce low confidence:
|
| 611 |
-
|
| 612 |
-
1. **Mixed Sentiment Examples:**
|
| 613 |
-
- "ลำไยอร่อยดีสดมากและลูกใหญ่ด้วยแต่เน่าไปครึ่งนึ..." → Positive (55.0%) or Question (52.1%)
|
| 614 |
-
- "เกือบดีแล้วล่ะ เหลือแค่ดีจริงๆ นิดเดียว" → Neutral (54.6%)
|
| 615 |
-
|
| 616 |
-
2. **Ambiguous Praise with Criticism:**
|
| 617 |
-
- "สุดยอดไปเลย! เธอใช้เวลาทั้งวันทำงานชิ้นนี้ออกม..." → Positive (51.7%)
|
| 618 |
-
|
| 619 |
-
**High Confidence Predictions (> 99%):**
|
| 620 |
-
The model excels at:
|
| 621 |
-
- Clear sarcasm: "เก่งจังเลยนะ ทำผิดซ้ำได้เหมือนเดิมเป๊ะเลย" → Negative (99.98%)
|
| 622 |
-
- Obvious negative sentiment: "ไม่ให้ดาวเลย" → Negative (99.94%)
|
| 623 |
-
- Simple positive expressions: "ให้5ดาวเลย" → Positive (99.98%)
|
| 624 |
|
| 625 |
#### Known Limitations
|
| 626 |
|
|
@@ -731,166 +550,6 @@ The model excels at:
|
|
| 731 |
|
| 732 |
### Best Practices for Implementation
|
| 733 |
|
| 734 |
-
#### Text Preprocessing
|
| 735 |
-
```python
|
| 736 |
-
def preprocess_thai_text(text):
|
| 737 |
-
"""
|
| 738 |
-
Recommended preprocessing for Thai text
|
| 739 |
-
"""
|
| 740 |
-
# Remove excessive whitespace
|
| 741 |
-
text = ' '.join(text.split())
|
| 742 |
-
|
| 743 |
-
# Handle common Thai punctuation
|
| 744 |
-
text = text.replace('...', ' ')
|
| 745 |
-
text = text.replace('!!', '!')
|
| 746 |
-
|
| 747 |
-
# Normalize quotation marks
|
| 748 |
-
text = text.replace('"', '"').replace('"', '"')
|
| 749 |
-
|
| 750 |
-
return text.strip()
|
| 751 |
-
```
|
| 752 |
-
|
| 753 |
-
#### Confidence Thresholding
|
| 754 |
-
```python
|
| 755 |
-
def classify_with_confidence(text, threshold=0.6):
|
| 756 |
-
"""
|
| 757 |
-
Classification with confidence thresholding
|
| 758 |
-
Recommended threshold: 0.6 based on empirical testing
|
| 759 |
-
"""
|
| 760 |
-
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
|
| 761 |
-
|
| 762 |
-
with torch.no_grad():
|
| 763 |
-
outputs = model(**inputs)
|
| 764 |
-
predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
|
| 765 |
-
confidence = torch.max(predictions).item()
|
| 766 |
-
predicted_class = torch.argmax(predictions, dim=-1).item()
|
| 767 |
-
|
| 768 |
-
if confidence >= threshold:
|
| 769 |
-
return labels[predicted_class], confidence
|
| 770 |
-
else:
|
| 771 |
-
return "Low Confidence - Manual Review Needed", confidence
|
| 772 |
-
|
| 773 |
-
# Enhanced classification with question detection fallback
|
| 774 |
-
def enhanced_classify(text, confidence_threshold=0.6):
|
| 775 |
-
"""
|
| 776 |
-
Enhanced classification with special handling for low confidence
|
| 777 |
-
and potential question detection
|
| 778 |
-
"""
|
| 779 |
-
sentiment, confidence = classify_with_confidence(text, confidence_threshold)
|
| 780 |
-
|
| 781 |
-
# Special handling for low confidence predictions
|
| 782 |
-
if confidence < confidence_threshold:
|
| 783 |
-
# Simple question detection fallback
|
| 784 |
-
question_indicators = ['?', 'ไหม', 'หรือ', 'ครับ', 'คะ', 'มั้ย']
|
| 785 |
-
if any(indicator in text for indicator in question_indicators):
|
| 786 |
-
return "Question (Detected by Rules)", confidence
|
| 787 |
-
else:
|
| 788 |
-
return f"Uncertain ({sentiment})", confidence
|
| 789 |
-
|
| 790 |
-
return sentiment, confidence
|
| 791 |
-
|
| 792 |
-
# Example usage with test cases
|
| 793 |
-
test_texts = [
|
| 794 |
-
"ลำไยอร่อยดีสดมากและลูกใหญ่ด้วยแต่เน่าไปครึ่งนึ...", # Mixed sentiment
|
| 795 |
-
"สุดยอดไปเลย! เธอใช้เวลาทั้งวันทำงานชิ้นนี้ออกม...", # Low confidence positive
|
| 796 |
-
"เก่งจังเลยนะ ทำผิดซ้ำ��ด้เหมือนเดิมเป๊ะเลย", # High confidence sarcasm
|
| 797 |
-
]
|
| 798 |
-
|
| 799 |
-
for text in test_texts:
|
| 800 |
-
result, conf = enhanced_classify(text)
|
| 801 |
-
print(f"Text: {text[:50]}...")
|
| 802 |
-
print(f"Result: {result} (Confidence: {conf:.1%})")
|
| 803 |
-
print()
|
| 804 |
-
```
|
| 805 |
-
|
| 806 |
-
#### Production Deployment Example
|
| 807 |
-
```python
|
| 808 |
-
from fastapi import FastAPI
|
| 809 |
-
from pydantic import BaseModel
|
| 810 |
-
import logging
|
| 811 |
-
|
| 812 |
-
app = FastAPI()
|
| 813 |
-
|
| 814 |
-
class SentimentRequest(BaseModel):
|
| 815 |
-
text: str
|
| 816 |
-
|
| 817 |
-
class SentimentResponse(BaseModel):
|
| 818 |
-
sentiment: str
|
| 819 |
-
confidence: float
|
| 820 |
-
warning: str = None
|
| 821 |
-
|
| 822 |
-
def classify_with_warnings(text, confidence_threshold=0.6):
|
| 823 |
-
"""
|
| 824 |
-
Production-ready classification with warnings for low confidence
|
| 825 |
-
"""
|
| 826 |
-
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
|
| 827 |
-
|
| 828 |
-
with torch.no_grad():
|
| 829 |
-
outputs = model(**inputs)
|
| 830 |
-
predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
|
| 831 |
-
confidence = torch.max(predictions).item()
|
| 832 |
-
predicted_class = torch.argmax(predictions, dim=-1).item()
|
| 833 |
-
|
| 834 |
-
sentiment = labels[predicted_class]
|
| 835 |
-
warning = None
|
| 836 |
-
|
| 837 |
-
# Add warnings based on empirical testing
|
| 838 |
-
if confidence < confidence_threshold:
|
| 839 |
-
warning = "Low confidence prediction - manual review recommended"
|
| 840 |
-
|
| 841 |
-
if predicted_class == 0: # Question class
|
| 842 |
-
warning = "Question classification has known accuracy issues - consider manual review"
|
| 843 |
-
|
| 844 |
-
# Detect potential mixed sentiment
|
| 845 |
-
if confidence < 0.7 and any(pos_word in text for pos_word in ['ดี', 'อร่อย', 'สวย']) and any(neg_word in text for neg_word in ['แย่', 'เน่า', 'แต่']):
|
| 846 |
-
warning = "Possible mixed sentiment detected - consider aspect-based analysis"
|
| 847 |
-
|
| 848 |
-
return sentiment, confidence, warning
|
| 849 |
-
|
| 850 |
-
@app.post("/analyze-sentiment", response_model=SentimentResponse)
|
| 851 |
-
async def analyze_sentiment(request: SentimentRequest):
|
| 852 |
-
try:
|
| 853 |
-
# Preprocess text
|
| 854 |
-
text = preprocess_thai_text(request.text)
|
| 855 |
-
|
| 856 |
-
# Get prediction with warnings
|
| 857 |
-
sentiment, confidence, warning = classify_with_warnings(text)
|
| 858 |
-
|
| 859 |
-
# Log low confidence predictions for monitoring
|
| 860 |
-
if confidence < 0.6:
|
| 861 |
-
logging.warning(f"Low confidence prediction: {text[:50]}... -> {sentiment} ({confidence:.3f})")
|
| 862 |
-
|
| 863 |
-
return SentimentResponse(
|
| 864 |
-
sentiment=sentiment,
|
| 865 |
-
confidence=confidence,
|
| 866 |
-
warning=warning
|
| 867 |
-
)
|
| 868 |
-
|
| 869 |
-
except Exception as e:
|
| 870 |
-
logging.error(f"Error processing text: {str(e)}")
|
| 871 |
-
return SentimentResponse(
|
| 872 |
-
sentiment="Error",
|
| 873 |
-
confidence=0.0,
|
| 874 |
-
warning="Processing error occurred"
|
| 875 |
-
)
|
| 876 |
-
|
| 877 |
-
# Batch processing endpoint for efficiency
|
| 878 |
-
@app.post("/analyze-batch")
|
| 879 |
-
async def analyze_batch(texts: list[str]):
|
| 880 |
-
"""
|
| 881 |
-
Batch processing for multiple texts
|
| 882 |
-
"""
|
| 883 |
-
results = []
|
| 884 |
-
for text in texts:
|
| 885 |
-
sentiment, confidence, warning = classify_with_warnings(text)
|
| 886 |
-
results.append({
|
| 887 |
-
"text": text[:100] + "..." if len(text) > 100 else text,
|
| 888 |
-
"sentiment": sentiment,
|
| 889 |
-
"confidence": confidence,
|
| 890 |
-
"warning": warning
|
| 891 |
-
})
|
| 892 |
-
return {"results": results}
|
| 893 |
-
```
|
| 894 |
|
| 895 |
## Citation
|
| 896 |
|
|
@@ -898,21 +557,15 @@ async def analyze_batch(texts: list[str]):
|
|
| 898 |
|
| 899 |
**BibTeX:**
|
| 900 |
```bibtex
|
| 901 |
-
@misc{MultiSent-E5,
|
| 902 |
title={Thai-sentiment-e5: A Fine-tuned Multilingual Sentiment Analysis Model for Thai Text Classification},
|
| 903 |
author={ZombitX64 and Janutsaha, Krittanut and Saengwichain, Chanyut},
|
| 904 |
year={2024},
|
| 905 |
-
url={https://huggingface.co/ZombitX64/MultiSent-E5},
|
| 906 |
note={Hugging Face Model Repository}
|
| 907 |
}
|
| 908 |
```
|
| 909 |
|
| 910 |
-
**APA Style:**
|
| 911 |
-
ZombitX64, Janutsaha, K., & Saengwichain, C. (2024). *MultiSent-E5: A Fine-tuned Multilingual Sentiment Analysis Model for Thai Text Classification*. Hugging Face. https://huggingface.co/ZombitX64/MultiSent-E5
|
| 912 |
-
|
| 913 |
-
**IEEE Style:**
|
| 914 |
-
ZombitX64, K. Janutsaha, and C. Saengwichain, "MultiSent-E5: A Fine-tuned Multilingual Sentiment Analysis Model for Thai Text Classification," Hugging Face, 2024. [Online]. Available: https://huggingface.co/ZombitX64/MultiSent-E5
|
| 915 |
-
|
| 916 |
### Usage in Publications
|
| 917 |
|
| 918 |
If you use this model in your research or applications, please cite both this model and the base model:
|
|
@@ -939,7 +592,7 @@ If you use this model in your research or applications, please cite both this mo
|
|
| 939 |
For questions, issues, or contributions regarding this model, please use the following channels:
|
| 940 |
|
| 941 |
* **Primary Contact:** Hugging Face model repository issues and discussions
|
| 942 |
-
* **Repository:** [https://huggingface.co/ZombitX64/MultiSent-E5](https://huggingface.co/ZombitX64/MultiSent-E5)
|
| 943 |
* **Community:** Hugging Face community forums for general questions
|
| 944 |
|
| 945 |
### Collaboration Opportunities
|
|
|
|
| 114 |
- ZombitX64/Sentiment-Benchmark
|
| 115 |
---
|
| 116 |
|
| 117 |
+
# ZombitX64-MultiSent-E5-Pro
|
| 118 |
|
| 119 |
<div align="center">
|
| 120 |
<picture>
|
|
|
|
| 221 |
import torch
|
| 222 |
|
| 223 |
# Load the model and tokenizer
|
| 224 |
+
model_name = "ZombitX64/MultiSent-E5-Pro"
|
| 225 |
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
| 226 |
model = AutoModelForSequenceClassification.from_pretrained(model_name)
|
| 227 |
|
|
|
|
| 280 |
|
| 281 |
# Create a sentiment analysis pipeline
|
| 282 |
classifier = pipeline("text-classification",
|
| 283 |
+
model="ZombitX64/MultiSent-E5-Pro",
|
| 284 |
+
tokenizer="ZombitX64/MultiSent-E5-Pro")
|
| 285 |
|
| 286 |
# Analyze sentiment
|
| 287 |
texts = [
|
|
|
|
| 364 |
- Early convergence suggests effective transfer learning from the base model
|
| 365 |
|
| 366 |
============================================================
|
| 367 |
+
Evaluating: ZombitX64/MultiSent-E5-Pro
|
| 368 |
============================================================
|
| 369 |
+
Loading ZombitX64/MultiSent-E5-Pro...
|
| 370 |
+
Predicting 2183 samples...
|
| 371 |
+
Predicting: 2183/2183
|
| 372 |
+
Accuracy: 0.846
|
| 373 |
+
F1-Macro: 0.846
|
| 374 |
+
F1-Weighted: 0.847
|
| 375 |
+
Avg Confidence: 0.985
|
| 376 |
+
Low Confidence %: 1.0%
|
| 377 |
+
Error Rate: 0.154
|
| 378 |
+
|
| 379 |
+
Sample Errors:
|
| 380 |
+
'今天的表现无可挑剔' -> neutral (conf: 1.00) [True: positive]
|
| 381 |
+
'这真是个天才的想法,我简直佩服得五体投地' -> positive (conf: 1.00) [True: negative]
|
| 382 |
+
'你真是太能干了,把事情搞成这样' -> positive (conf: 1.00) [True: negative]
|
| 383 |
+
'这个项目真是太成功了,成功到一塌糊涂' -> positive (conf: 1.00) [True: negative]
|
| 384 |
+
'这饭菜做得真是太好吃了,我一点都吃不下' -> positive (conf: 1.00) [True: negative]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 385 |
|
| 386 |
+
============================================================
|
| 387 |
+
BEST PERFORMING MODEL: ZombitX64/MultiSent-E5-Pro
|
| 388 |
+
============================================================
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 389 |
|
| 390 |
+
Per-Class Performance:
|
| 391 |
+
precision recall f1-score support
|
| 392 |
+
negative 0.910 0.846 0.877 661.0
|
| 393 |
+
neutral 0.719 0.816 0.764 517.0
|
| 394 |
+
positive 0.830 0.943 0.883 471.0
|
| 395 |
+
question 0.944 0.790 0.860 534.0
|
| 396 |
+
|
| 397 |
+
================================================================================
|
| 398 |
+
COMPREHENSIVE MODEL COMPARISON REPORT
|
| 399 |
+
Dataset: ZombitX64/Sentiment-Benchmark
|
| 400 |
+
================================================================================
|
| 401 |
+
|
| 402 |
+
Ranked by F1-Macro Score:
|
| 403 |
+
Model Accuracy F1-Macro F1-Weighted Avg_Confidence Low_Conf_% Error_Rate
|
| 404 |
+
ZombitX64/MultiSent-E5-Pro 0.8461 0.8461 0.8475 0.9853 0.9620 0.1539
|
| 405 |
+
ZombitX64/MultiSent-E5 0.8062 0.8062 0.8072 0.9708 1.6033 0.1938
|
| 406 |
+
ZombitX64/sentiment-103 0.5740 0.4987 0.5020 0.9647 2.2446 0.4260
|
| 407 |
+
ZombitX64/Sentiment-03 0.4828 0.4906 0.4856 0.9609 2.7485 0.5172
|
| 408 |
+
ZombitX64/Sentiment-02 0.4137 0.3884 0.3910 0.8151 10.0779 0.5863
|
| 409 |
+
ZombitX64/Thai-sentiment-e5 0.4961 0.3713 0.3704 0.9874 0.8246 0.5039
|
| 410 |
+
nlptown/bert-base-multilingual-uncased-sentiment 0.3587 0.2870 0.2896 0.4103 87.9066 0.6413
|
| 411 |
+
ZombitX64/Sentiment-01 0.2712 0.1928 0.1894 0.5085 94.5946 0.7288
|
| 412 |
+
SandboxBhh/sentiment-thai-text-model 0.2620 0.1807 0.1982 0.8610 20.2016 0.7380
|
| 413 |
+
Thaweewat/wangchanberta-hyperopt-sentiment-01 0.2336 0.1501 0.1655 0.9128 2.9776 0.7664
|
| 414 |
+
phoner45/wangchan-sentiment-thai-text-model 0.2203 0.1073 0.1270 0.7123 41.7316 0.7797
|
| 415 |
+
poom-sci/WangchanBERTa-finetuned-sentiment 0.2093 0.1061 0.1246 0.7889 14.7045 0.7907
|
| 416 |
+
cardiffnlp/twitter-xlm-roberta-base-sentiment 0.0944 0.0848 0.0841 0.6897 32.2492 0.9056
|
| 417 |
|
| 418 |
|
| 419 |
|
|
|
|
| 423 |
|
| 424 |
The model was evaluated on a carefully selected validation set with the following characteristics:
|
| 425 |
|
| 426 |
+
* **Total Samples:** 2183
|
| 427 |
* **Selection Method:** Stratified random sampling to maintain class distribution
|
| 428 |
* **Data Quality:** Manually verified and cleaned validation samples
|
| 429 |
* **Evaluation Period:** Final model checkpoint from epoch 5
|
|
|
|
| 440 |
- **Recall:** Per-class and overall recall scores
|
| 441 |
- **Support:** Number of samples per class in validation set
|
| 442 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 443 |
|
| 444 |
#### Known Limitations
|
| 445 |
|
|
|
|
| 550 |
|
| 551 |
### Best Practices for Implementation
|
| 552 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 553 |
|
| 554 |
## Citation
|
| 555 |
|
|
|
|
| 557 |
|
| 558 |
**BibTeX:**
|
| 559 |
```bibtex
|
| 560 |
+
@misc{MultiSent-E5-Pro,
|
| 561 |
title={Thai-sentiment-e5: A Fine-tuned Multilingual Sentiment Analysis Model for Thai Text Classification},
|
| 562 |
author={ZombitX64 and Janutsaha, Krittanut and Saengwichain, Chanyut},
|
| 563 |
year={2024},
|
| 564 |
+
url={https://huggingface.co/ZombitX64/MultiSent-E5-Pro},
|
| 565 |
note={Hugging Face Model Repository}
|
| 566 |
}
|
| 567 |
```
|
| 568 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 569 |
### Usage in Publications
|
| 570 |
|
| 571 |
If you use this model in your research or applications, please cite both this model and the base model:
|
|
|
|
| 592 |
For questions, issues, or contributions regarding this model, please use the following channels:
|
| 593 |
|
| 594 |
* **Primary Contact:** Hugging Face model repository issues and discussions
|
| 595 |
+
* **Repository:** [https://huggingface.co/ZombitX64/MultiSent-E5-Pro](https://huggingface.co/ZombitX64/MultiSent-E5-Pro)
|
| 596 |
* **Community:** Hugging Face community forums for general questions
|
| 597 |
|
| 598 |
### Collaboration Opportunities
|