docs/GPT_METRICS_CALCULATION.md

GPT Labeling Approach - Main Metrics Calculation

Overview

The RAG evaluation system uses GPT-based labeling to calculate four TRACE metrics. The GPT model analyzes responses sentence-by-sentence and identifies which parts are supported by retrieved documents, enabling precise metric calculation.

---

The Four TRACE Metrics

1. Context Relevance (R) - What's Actually Relevant?

Definition: Fraction of retrieved context that is relevant to answering the user's question.

Calculation:

Context Relevance = Number of relevant sentences / Total retrieved sentences
Normalized: min(1.0, count / 20)

Formula:

R = |Relevant Sentences| / |Total Sentences|

Data Source from GPT:

gpt_labels.all_relevant_sentence_keys
# List of document sentence keys identified as relevant
# Example: ["0a", "0b", "1c", "2a"]

Example:

Retrieved 30 sentences total
GPT identifies 12 as relevant to question "What is machine learning?"
Context Relevance = 12/30 = 0.40 (40%)

What It Tells You:

• ✓ How good was the retrieval?
• ✓ Did we pull documents about the right topic?
• ✓ Are there irrelevant documents in the results?

---

2. Context Utilization (T) - How Much Was Used?

Definition: Fraction of relevant context that the response actually used to generate its answer.

Calculation:

Context Utilization = Number of utilized sentences / Number of relevant sentences

Formula:

U = |Utilized Sentences| / |Relevant Sentences|

Data Source from GPT:

gpt_labels.all_utilized_sentence_keys
# List of document sentence keys actually used in response
# Example: ["0a", "0b", "1c"]

Example:

Context Relevance found 12 relevant sentences: ["0a", "0b", "1a", "1c", "2a", ...]
GPT identifies 8 actually used in response: ["0a", "0b", "1c", "2a", ...]
Context Utilization = 8/12 = 0.67 (67%)

What It Tells You:

• ✓ Did the LLM actually use the available information?
• ✓ Is the response limited by context availability?
• ✓ Is context being ignored/wasted?

Problem Pattern:

• High Relevance (0.9) + Low Utilization (0.3) → Retrieval is good, but LLM isn't using it → Fix: Improve prompt instructions

---

3. Completeness (C) - Was It Comprehensive?

Definition: Fraction of relevant information that is covered by the response.

Calculation:

Completeness = (Relevant ∩ Utilized) / Relevant
            = Relevant sentences that were used / All relevant sentences

Formula:

C = |Relevant ∩ Utilized| / |Relevant|

Data Source from GPT:

# Set intersection:
relevant_set = set(gpt_labels.all_relevant_sentence_keys)
utilized_set = set(gpt_labels.all_utilized_sentence_keys)
intersection = relevant_set & utilized_set

completeness = len(intersection) / len(relevant_set)

Example:

Relevant sentences: {"0a", "0b", "1a", "1c", "2a", "2b", "3a"}  (7 total)
Utilized sentences: {"0a", "0b", "1c", "2a"}                    (4 used)

Overlap (Relevant AND Used): {"0a", "0b", "1c", "2a"}            (4 in both)
Completeness = 4/7 = 0.57 (57%)

Missing: "1a", "2b", "3a" were relevant but not mentioned

What It Tells You:

• ✓ Did the response cover all important information?
• ✓ What relevant details were omitted?
• ✓ Is the response comprehensive?

Problem Pattern:

• Low Completeness (0.4) with High Adherence (1.0) → Response is accurate but limited → Missing important information → Fix: Improve retrieval coverage or summarization

---

4. Adherence (A) - Was It Grounded?

Definition: Whether the response is fully grounded in the retrieved context (no hallucinations).

Calculation:

Adherence = 1.0 if ALL sentences are fully supported, 0.0 otherwise
           (Boolean: fully grounded or contains hallucination)

Formula:

A = 1.0 if all(sentence.fully_supported for all sentences) else 0.0

Data Source from GPT:

gpt_labels.sentence_support_information
# For each response sentence:
# {
#   "response_sentence_key": "a",
#   "fully_supported": true/false,    # ← This determines adherence
#   "supporting_sentence_keys": ["0a", "0b"],
#   "explanation": "..."
# }

fully_supported_count = sum(
    1 for s in sentence_support_information
    if s.get("fully_supported", False)
)

adherence = 1.0 if fully_supported_count == total_sentences else 0.0

Example 1 - Perfect Adherence:

Response sentences:
  a. "Machine learning is a subset of AI."
     └─ Fully supported by document 0a ✓
  b. "It uses algorithms to learn from data."
     └─ Fully supported by document 1b ✓
  c. "Common applications include image recognition."
     └─ Fully supported by documents 2a, 2b ✓

ALL sentences fully supported → Adherence = 1.0 (100%)

Example 2 - Contains Hallucination:

Response sentences:
  a. "Machine learning is a subset of AI."
     └─ Fully supported by document 0a ✓
  b. "It requires quantum computers."
     └─ NOT supported by any document ✗ (Hallucination!)
  c. "Common applications include image recognition."
     └─ Fully supported by documents 2a, 2b ✓

ONE sentence NOT fully supported → Adherence = 0.0 (0%)

What It Tells You:

• ✓ Is the response truthful/grounded?
• ✓ Does it contain hallucinations?
• ✓ Can we trust the answer?

---

How GPT Labeling Works

Step 1: Sentencization

Documents:

Document 0: "Machine learning is AI. It learns from data."
Document 1: "Neural networks are models. They mimic brains."

↓ Splits into sentences with keys:

0a: "Machine learning is AI."
0b: "It learns from data."
1a: "Neural networks are models."
1b: "They mimic brains."

Response:

"Machine learning uses neural networks. They learn patterns."

↓ Splits into sentences:

a: "Machine learning uses neural networks."
b: "They learn patterns."

Step 2: GPT Labeling

GPT analyzes and identifies:

1. Relevance: Which document sentences are relevant to the question?
2. Utilization: Which document sentences were actually used in the response?
3. Support: Is each response sentence fully/partially/not supported?

GPT Output (JSON):

{
  "relevance_explanation": "Document discusses ML basics...",
  "all_relevant_sentence_keys": ["0a", "0b", "1a"],
  "overall_supported_explanation": "Response is grounded...",
  "overall_supported": true,
  "sentence_support_information": [
    {
      "response_sentence_key": "a",
      "explanation": "Matches document sentences...",
      "supporting_sentence_keys": ["0a", "1a"],
      "fully_supported": true
    },
    {
      "response_sentence_key": "b",
      "explanation": "Partially supported...",
      "supporting_sentence_keys": ["1b"],
      "fully_supported": false
    }
  ],
  "all_utilized_sentence_keys": ["0a", "1a", "1b"]
}

Step 3: Metric Calculation

Relevant: ["0a", "0b", "1a"]              (3 sentences)
Utilized: ["0a", "1a", "1b"]              (3 sentences)

Context Relevance = |["0a", "0b", "1a"]| / total_sentences
                  = 3 / ~20 = 0.15

Context Utilization = |Utilized| / |Relevant|
                    = 3 / 3 = 1.0

Completeness = |Relevant ∩ Utilized| / |Relevant|
             = |{"0a", "1a"}| / |{"0a", "0b", "1a"}|
             = 2 / 3 = 0.67

Adherence = All sentences fully supported? → Check each
          = sentence_a (true) AND sentence_b (false)
          = false → 0.0

---

Code Implementation

Metric Calculation Methods

def _compute_context_relevance(self, gpt_labels: GPTLabelingOutput) -> float:
    """Count relevant sentences, normalize to 0-1."""
    if not gpt_labels.all_relevant_sentence_keys:
        return 0.0
    return min(1.0, len(gpt_labels.all_relevant_sentence_keys) / 20.0)

def _compute_context_utilization(self, gpt_labels: GPTLabelingOutput) -> float:
    """Utilized / Relevant."""
    relevant_count = len(gpt_labels.all_relevant_sentence_keys)
    utilized_count = len(gpt_labels.all_utilized_sentence_keys)
    if relevant_count == 0:
        return 0.0
    return min(1.0, utilized_count / relevant_count)

def _compute_completeness(self, gpt_labels: GPTLabelingOutput, 
                        ground_truth: Optional[str] = None) -> float:
    """(Relevant AND Utilized) / Relevant."""
    relevant_set = set(gpt_labels.all_relevant_sentence_keys)
    utilized_set = set(gpt_labels.all_utilized_sentence_keys)
    intersection = len(relevant_set & utilized_set)
    if len(relevant_set) == 0:
        return 1.0 if len(utilized_set) == 0 else 0.0
    return intersection / len(relevant_set)

def _compute_adherence(self, gpt_labels: GPTLabelingOutput) -> float:
    """All sentences fully supported? Boolean: 1.0 or 0.0."""
    total_sentences = len(gpt_labels.sentence_support_information)
    if total_sentences == 0:
        return 1.0
    fully_supported_count = sum(
        1 for s in gpt_labels.sentence_support_information
        if s.get("fully_supported", False)
    )
    return 1.0 if fully_supported_count == total_sentences else 0.0

---

Complete Example: Full Calculation

Input

Question: "What is machine learning?"

Retrieved Documents:

Doc 0: "Machine learning is a subset of AI. It learns patterns from data. 
        Algorithms improve through experience."
Doc 1: "Deep learning uses neural networks. It's popular in computer vision."
Doc 2: "Supervised learning needs labeled data. Unsupervised learning finds patterns."

LLM Response:

"Machine learning is a field of AI that learns from data. Deep learning 
uses neural networks. It's powerful for image recognition."

GPT Labeling Process

Sentencized Documents:

0a: "Machine learning is a subset of AI."
0b: "It learns patterns from data."
0c: "Algorithms improve through experience."
1a: "Deep learning uses neural networks."
1b: "It's popular in computer vision."
2a: "Supervised learning needs labeled data."
2b: "Unsupervised learning finds patterns."

Sentencized Response:

a: "Machine learning is a field of AI that learns from data."
b: "Deep learning uses neural networks."
c: "It's powerful for image recognition."

GPT Analysis:

{
  "all_relevant_sentence_keys": ["0a", "0b", "1a", "1b"],
  "all_utilized_sentence_keys": ["0a", "0b", "1a", "1b"],
  "sentence_support_information": [
    {
      "response_sentence_key": "a",
      "supporting_sentence_keys": ["0a", "0b"],
      "fully_supported": true
    },
    {
      "response_sentence_key": "b",
      "supporting_sentence_keys": ["1a"],
      "fully_supported": true
    },
    {
      "response_sentence_key": "c",
      "supporting_sentence_keys": ["1b"],
      "fully_supported": false  // "powerful for image recognition" not explicitly in docs
    }
  ]
}

Metric Calculation

Relevant: ["0a", "0b", "1a", "1b"]       (4 sentences)
Utilized: ["0a", "0b", "1a", "1b"]       (4 sentences)
Total sentences retrieved: 7

Context Relevance = 4 / 20 = 0.20 (20%)
  └─ 4 out of ~20 average sentences are relevant

Context Utilization = 4 / 4 = 1.0 (100%)
  └─ All relevant information was used

Completeness = |{"0a","0b","1a","1b"} ∩ {"0a","0b","1a","1b"}| / 4
             = 4 / 4 = 1.0 (100%)
  └─ All relevant info was covered

Adherence = All fully supported?
          = sentence_a (true) AND sentence_b (true) AND sentence_c (false)
          = false → 0.0 (0%)
  └─ Contains hallucination about "powerful for image recognition"

Average Score = (0.20 + 1.0 + 1.0 + 0.0) / 4 = 0.55

---

Key Insights

1. Complementary Metrics

Metric	Measures	Ideal Value
Relevance	Quality of retrieval	High (0.7+)
Utilization	LLM uses available info	High (0.7+)
Completeness	Coverage of information	High (0.7+)
Adherence	Grounding (no hallucination)	Perfect (1.0)

2. Common Patterns

Pattern 1: Good Retrieval, Bad Generation

Relevance: 0.85 (good retrieval)
Utilization: 0.40 (not using it)
→ Problem: LLM not leveraging context
→ Fix: Improve prompt instructions

Pattern 2: Conservative but Accurate

Completeness: 0.50 (missing info)
Adherence: 1.0 (all correct)
→ Problem: Limited but grounded response
→ Fix: Improve retrieval coverage

Pattern 3: Comprehensive and Grounded

Relevance: 0.75, Utilization: 0.80, Completeness: 0.85, Adherence: 1.0
→ Excellent RAG system
→ Action: Monitor and maintain

3. Mathematical Relationships

Completeness ≤ Utilization ≤ Relevance
(Because: utilization requires relevance, completeness requires utilization)

Also:
If Relevance = 0 → Utilization = 0, Completeness = 0
If Utilization = 0 → Completeness = 0 (but Relevance can be > 0)

---

Advantages of GPT Labeling

✅ Semantic Understanding

• Not just keyword matching
• Understands meaning and context
• Detects subtle hallucinations

✅ Fine-Grained Analysis

• Sentence-level support mapping
• Identifies exactly which info is supported
• Pinpoints problematic claims

✅ Comprehensive

• Evaluates all four TRACE metrics
• Single pass through documents
• Complete audit trail

✅ Transparent

• Full explanation for each metric
• Shows supporting evidence
• Human-verifiable results

---

Limitations

❌ Cost

• API calls per evaluation (~2.5s per eval with rate limiting)
• At 30 RPM: 50 evals = 3-5 minutes

❌ Semantic Brittleness

• Depends on GPT's understanding
• May miss implicit knowledge
• Sensitive to phrasing

❌ Normalization

• Context Relevance normalized by 20 (arbitrary baseline)
• Different domain sizes affect scaling

❌ Binary Adherence

• One hallucination = 0.0 adherence
• No partial credit for mostly correct

---

Summary

The GPT labeling approach calculates TRACE metrics by:

1. Splitting documents and response into sentences
2. Analyzing with GPT to identify relevant/utilized/supported information
3. Computing metrics from the labeled sentence keys:
   • Relevance: What's relevant in retrieved docs?
   • Utilization: What of that was actually used?
   • Completeness: What coverage of relevant info?
   • Adherence: Is all information grounded?

This enables precise, interpretable evaluation of RAG system quality.