backend/docs/efficiency_guide.md

BGE-M3 Efficiency Guide

This guide explains how to measure and optimize the efficiency of the BAAI/bge-m3 model used in the IRIS project.

1. Key Metrics

Performance (Infrastructure)

• Latency: Time taken to generate an embedding for a single text. Critical for real-time search.
• Throughput: Number of documents processed per second. Important for batch processing (e.g., initial profile indexing).
• VRAM/RAM Usage: Memory footprint of the model. BGE-M3 is ~2.2GB in FP32.

Retrieval Quality (Accuracy)

• Precision@K: The proportion of relevant candidates in the top K results.
  • Example: If you return 10 candidates and 3 are actually qualified, Precision@10 = 30%.
• Recall@K (Correlation to User's "callback"): The proportion of total relevant candidates that were successfully captured in the top K.
  • Example: If there are 5 qualified candidates in the database and your search finds 4 of them in the top 10, Recall@10 = 80%.
• MRR (Mean Reciprocal Rank): Evaluates how high the first relevant candidate is ranked.
  • Formula: $1 / Rank$. If the best candidate is at position #1, score is 1.0. If at #2, score is 0.5.
• NDCG (Normalized Discounted Cumulative Gain): Measures the overall quality of the ranking order, giving more weight to highly relevant results at the very top.

2. BGE-M3 Specific Features

BGE-M3 is a "multi-function" model. You can measure efficiency across three modes:

1. Dense Retrieval: Standard 1024d vectors. Fast and semantic.
2. Sparse Retrieval (Lexical): Similar to BM25 but learned. More efficient for exact keyword matching.
3. Multi-Vector (ColBERT style): Most accurate but highest storage and latency cost.

3. Optimization Techniques

Precision Tuning

• FP16: Use model.half() if on GPU to double speed and halve memory with negligible accuracy loss.
• Quantization: Int8 or GGUF formats can reduce memory usage by 4x.

Batching

Using optimal batch sizes (e.g., 16-32) significantly improves throughput compared to single-sentence processing.

4. Measuring Quality in IRIS

To measure quality, create a "Golden Dataset" of (Job Description, Relevant Profiles) and calculate Hit Rate:

1. Fetch top 10 profiles for a job.
2. Check if the "ideal" candidate is in that list.
3. Average this over 50 test cases.