README.md

---
language: 
- vi
- en
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- mathematics
- vietnamese
- binary-classification
- hard-negatives
- loss-based-early-stopping
- e5-base
- exact-retrieval
base_model: intfloat/multilingual-e5-base
metrics:
- mean_reciprocal_rank
- hit_rate
- accuracy
datasets:
- custom-vietnamese-math
---

# E5-Math-Vietnamese-Binary: Hard Negatives + Loss-based Early Stopping

## Model Overview

Fine-tuned E5-base model optimized for **exact chunk retrieval** in Vietnamese mathematics using:
- **🎯 Binary Classification**: Correct vs Incorrect (instead of 3-level hierarchy)
- **💪 Hard Negatives**: Related chunks as hard negatives for better discrimination  
- **⏰ Loss-based Early Stopping**: Stops when validation loss stops improving
- **📊 Comprehensive Evaluation**: Hit@K, Accuracy@1, MRR metrics

## Performance Summary

### Training Results
- **Best Validation Loss**: N/A
- **Training Epochs**: 10
- **Early Stopping**: ❌ Not triggered
- **Training Time**: 4661.226378917694

### Test Performance 🌟 EXCELLENT
Outstanding performance with correct chunks consistently at top positions

| Metric | Base E5 | Fine-tuned | Improvement |
|--------|---------|------------|-------------|
| **MRR** | 0.7740 | 0.8505 | +0.0765 |
| **Accuracy@1** | 0.6129 | 0.7634 | +0.1505 |
| **Hit@1** | 0.6129 | 0.7634 | +0.1505 |
| **Hit@3** | 0.9462 | 0.9247 | -0.0215 |
| **Hit@5** | 1.0000 | 0.9785 | -0.0215 |

**Total Test Queries**: 93

## Key Innovations

### 🎯 Binary Classification Approach
Instead of traditional 3-level hierarchy (correct/related/irrelevant), this model uses:
- **Correct chunks**: Score 1.0 (positive examples)
- **Incorrect chunks**: Score 0.0 (includes both related and irrelevant)
- **Hard negatives**: Related chunks serve as challenging negative examples

### 💪 Hard Negatives Strategy
```python
# Training strategy
positive = correct_chunk           # Score: 1.0
hard_negative = related_chunk      # Score: 0.0 (but semantically close)
easy_negative = irrelevant_chunk   # Score: 0.0 (semantically distant)

# This forces model to learn fine-grained distinctions
```

### ⏰ Loss-based Early Stopping
- Monitors **validation loss** instead of MRR
- Stops when loss stops decreasing (patience=3)
- Prevents overfitting and saves training time

## Usage

### Basic Usage
```python
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

# Load model
model = SentenceTransformer('ThanhLe0125/ebd-math')

# ⚠️ CRITICAL: Must use E5 prefixes
query = "query: Định nghĩa hàm số đồng biến là gì?"
chunks = [
    "passage: Hàm số đồng biến trên khoảng (a;b) là hàm số mà...",  # Should rank #1
    "passage: Ví dụ về hàm số đồng biến: f(x) = 2x + 1...",         # Related (trained as hard negative)
    "passage: Phương trình bậc hai có dạng ax² + bx + c = 0"        # Irrelevant
]

# Encode and rank
query_emb = model.encode([query])
chunk_embs = model.encode(chunks)
similarities = cosine_similarity(query_emb, chunk_embs)[0]

# Get rankings
ranked_indices = similarities.argsort()[::-1]
for rank, idx in enumerate(ranked_indices, 1):
    print(f"Rank {rank}: Score {similarities[idx]:.4f} - {chunks[idx][:50]}...")
```

### Advanced Usage with Multiple Queries
```python
def find_best_chunks(queries, chunk_pool, top_k=3):
    """Find best chunks for multiple queries"""
    results = []
    
    for query in queries:
        # Ensure E5 format
        formatted_query = f"query: {query}" if not query.startswith("query:") else query
        formatted_chunks = [f"passage: {chunk}" if not chunk.startswith("passage:") else chunk 
                          for chunk in chunk_pool]
        
        # Encode
        query_emb = model.encode([formatted_query])
        chunk_embs = model.encode(formatted_chunks)
        similarities = cosine_similarity(query_emb, chunk_embs)[0]
        
        # Get top K
        top_indices = similarities.argsort()[::-1][:top_k]
        top_chunks = [
            {
                'chunk': chunk_pool[i],
                'similarity': similarities[i],
                'rank': rank + 1
            }
            for rank, i in enumerate(top_indices)
        ]
        
        results.append({
            'query': query,
            'top_chunks': top_chunks
        })
    
    return results

# Example
queries = [
    "Công thức tính đạo hàm của hàm hợp",
    "Cách giải phương trình bậc hai", 
    "Định nghĩa giới hạn của hàm số"
]

chunk_pool = [
    "Đạo hàm của hàm hợp: (f(g(x)))' = f'(g(x)) × g'(x)",
    "Giải phương trình bậc hai bằng công thức nghiệm",
    "Giới hạn của hàm số tại một điểm",
    # ... more chunks
]

results = find_best_chunks(queries, chunk_pool, top_k=3)
```

## Training Details

### Dataset
- **Domain**: Vietnamese mathematics education
- **Split**: Train/Validation/Test with proper separation
- **Hard Negatives**: Related mathematical concepts as challenging negatives
- **Easy Negatives**: Unrelated mathematical concepts

### Training Configuration
```python
Config:
    base_model = "intfloat/multilingual-e5-base"
    train_batch_size = 4
    learning_rate = 2e-5
    max_epochs = 10
    early_stopping_patience = 3
    loss_function = "MultipleNegativesRankingLoss"
    evaluation_metric = "validation_loss"
```

### Evaluation Methodology
1. **Training**: Binary classification with hard negatives
2. **Validation**: Loss-based monitoring for early stopping  
3. **Testing**: Comprehensive evaluation with restored 3-level hierarchy
4. **Metrics**: Hit@K, Accuracy@1, MRR comparison vs base model

## Model Architecture
- **Base**: intfloat/multilingual-e5-base
- **Max Sequence Length**: 256 tokens
- **Output Dimension**: 768
- **Similarity**: Cosine similarity
- **Training Loss**: MultipleNegativesRankingLoss

## Use Cases
- ✅ **Educational Q&A**: Find exact mathematical definitions and explanations
- ✅ **Content Retrieval**: Precise chunk retrieval for Vietnamese math content  
- ✅ **Tutoring Systems**: Quick and accurate answer finding
- ✅ **Knowledge Base Search**: Efficient mathematical concept lookup

## Performance Interpretation
- **Hit@1 ≥ 0.7**: 🌟 Excellent - Correct answer usually at #1
- **Hit@3 ≥ 0.8**: 🎯 Very Good - Correct answer in top 3
- **MRR ≥ 0.7**: 👍 Good - Low average rank for correct answers
- **Accuracy@1 ≥ 0.6**: ✅ Solid - Good precision for top result

## Limitations
- **Vietnamese-specific**: Optimized for Vietnamese mathematical terminology
- **Domain-specific**: Best performance on educational math content
- **Sequence length**: Limited to 256 tokens
- **E5 format required**: Must use "query:" and "passage:" prefixes

## Citation
```bibtex
@model{e5-math-vietnamese-binary,
  title={E5-Math-Vietnamese-Binary: Hard Negatives Fine-tuning for Mathematical Retrieval},
  author={ThanhLe0125},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/ThanhLe0125/ebd-math}
}
```

---
*Trained on July 01, 2025 using hard negatives and loss-based early stopping for optimal retrieval performance.*