EXECUTION_PLAN.md

Benchmark Data Collection & Vector DB Build Plan

Status: Data fetched, ready for vector DB integration
Date: October 19, 2025

---

✅ What We've Accomplished

1. Infrastructure Built

• ✅ Vector DB system (benchmark_vector_db.py)
• ✅ Data fetcher (fetch_benchmark_data.py)
• ✅ Post-processor (postprocess_benchmark_data.py)
• ✅ MCP tool integration (togmal_check_prompt_difficulty)

2. Data Collected

Total Questions: 500 MMLU-Pro questions
Source: TIGER-Lab/MMLU-Pro (test split)
Domains: 14 domains (math, physics, biology, health, law, etc.)
Sampling: Stratified across domains

Files Created:

• ./data/benchmark_results/raw_benchmark_results.json (500 questions)
• ./data/benchmark_results/collection_statistics.json

---

🎯 Current Situation

What Worked

✅ MMLU-Pro: 500 questions fetched successfully
✅ Stratified sampling: Balanced across 14 domains
✅ Infrastructure: All code ready for production

What Didn't Work

❌ GPQA Diamond: Gated dataset (needs HuggingFace auth)
❌ MATH dataset: Dataset name changed/moved on HuggingFace
❌ Per-question model results: OpenLLM Leaderboard doesn't expose detailed per-question results publicly

Key Finding

OpenLLM Leaderboard doesn't provide per-question results in downloadable datasets.

The open-llm-leaderboard/details_* datasets don't exist or aren't publicly accessible. We need an alternative approach.

---

🔄 Revised Strategy

Since we can't get real per-question success rates from leaderboards, we have 3 options:

Option A: Use Benchmark-Level Estimates (FAST - Recommended)

Time: Immediate
Accuracy: Good enough for MVP

Assign success rates based on published benchmark scores:

# From published leaderboard scores
BENCHMARK_SUCCESS_RATES = {
    "MMLU_Pro": {
        "physics": 0.52,
        "mathematics": 0.48,
        "biology": 0.55,
        "health": 0.58,
        "law": 0.62,
        # ... per domain
    }
}

Pros:

• ✅ Immediate deployment
• ✅ Based on real benchmark scores
• ✅ Good enough for capability boundary detection

Cons:

• ❌ No per-question granularity
• ❌ All questions in a domain get same score

Option B: Run Evaluations Ourselves (ACCURATE)

Time: 2-3 days
Cost: ~$50-100 API costs
Accuracy: Perfect

Run top 3-5 models on our 500 questions:

# Use llm-eval frameworks
pip install lm-eval-harness
lm-eval --model hf \
        --model_args pretrained=meta-llama/Meta-Llama-3.1-70B-Instruct \
        --tasks mmlu_pro \
        --output_path ./results/

Pros:

• ✅ Real per-question success rates
• ✅ Full control over which models
• ✅ Most accurate

Cons:

• ❌ Takes 2-3 days to run
• ❌ Requires GPU access or API costs
• ❌ Complex setup

Option C: Use Alternative Datasets with Known Difficulty (HYBRID)

Time: 1 day
Accuracy: Good

Use datasets that already have difficulty labels:

• ARC-Challenge: Has difficulty field
• CommonsenseQA: Has difficulty ratings
• TruthfulQA: Inherently hard (known low success)

Pros:

• ✅ Difficulty already labeled
• ✅ No need to run evaluations
• ✅ Quick to implement

Cons:

• ❌ Different benchmarks than MMLU-Pro/GPQA
• ❌ May not align with our use case

---

📊 Recommended Path Forward

Phase 1: Quick MVP (TODAY)

Use Option A - Benchmark-Level Estimates

1. Assign domain-level success rates based on published scores
2. Add variance within domains (±10%) for realism
3. Build vector DB with 500 questions
4. Test MCP tool with real prompts

Implementation:

# In benchmark_vector_db.py
DOMAIN_SUCCESS_RATES = {
    "mathematics": 0.48,
    "physics": 0.52,
    "chemistry": 0.54,
    "biology": 0.55,
    "health": 0.58,
    "law": 0.62,
    # Add small random variance per question
}

Timeline: 2 hours
Output: Working vector DB with 500 questions

Phase 2: Scale Up (THIS WEEK)

Expand to 1000+ questions

1. Authenticate with HuggingFace → access GPQA Diamond (200 questions)
2. Find MATH dataset alternative (lighteval/MATH-500 or similar)
3. Add ARC-Challenge (1000 questions with difficulty labels)

Timeline: 2-3 days
Output: 1000+ questions across multiple benchmarks

Phase 3: Real Evaluations (NEXT WEEK - Optional)

Run evaluations for perfect accuracy

1. Select top 3 models: Llama 3.1 70B, Qwen 2.5 72B, Claude 3.5
2. Run on our curated dataset (1000 questions)
3. Compute real success rates per question

Timeline: 3-5 days (depends on GPU access)
Output: Perfect per-question success rates

---

🚀 Immediate Next Steps (Option A)

Step 1: Update Vector DB with Domain Estimates

# Edit benchmark_vector_db.py to use domain-level success rates
cd /Users/hetalksinmaths/togmal

Step 2: Build Vector DB

python benchmark_vector_db.py
# Will index 500 MMLU-Pro questions with estimated success rates

Step 3: Test with Real Prompts

python test_vector_db.py

Step 4: Integrate with MCP Server

python togmal_mcp.py
# Tool: togmal_check_prompt_difficulty now works!

---

📈 Success Metrics

For MVP (Phase 1)

• 500+ questions indexed
• Domain-level success rates assigned
• Vector DB operational (<50ms queries)
• MCP tool tested with 10+ prompts
• Correctly identifies hard vs easy domains

For Scale (Phase 2)

• 1000+ questions indexed
• 3+ benchmarks represented
• Real difficulty labels (from GPQA/ARC)
• Stratified by low/medium/high success

For Production (Phase 3)

• Real per-question success rates
• 3+ top models evaluated
• Validated against known hard questions
• Integrated into Aqumen pipeline

---

💡 Key Insights

What We Learned

1. OpenLLM Leaderboard data isn't publicly queryable - we need to run evals ourselves or use estimates
2. MMLU-Pro has great coverage - 14 domains, 12K questions available
3. GPQA is gated but accessible - just need HuggingFace authentication
4. Vector similarity works well - even with 70 questions, domain matching was accurate

Strategic Decision

Start with estimates (Option A), validate with real evals (Option B) later

This gives us:

• ✅ Fast deployment: Working today
• ✅ Real validation: Can improve accuracy later
• ✅ Iterative approach: Learn from MVP before investing in evals

---

📝 Action Items

For You (Immediate)

1. Decide: Option A (estimates) or Option B (run evals)?
2. If Option A: Approve domain-level success rate estimates
3. If Option B: Decide which models to evaluate (API access needed)

For Me (Next)

1. Implement chosen option (1-2 hours for A, 2-3 days for B)
2. Build vector DB with 500 questions
3. Test MCP tool with real prompts
4. Document results in VECTOR_DB_STATUS.md

---

🎯 Recommendation

Go with Option A (Benchmark-Level Estimates) NOW

Rationale:

• Gets you a working system today
• Good enough for initial VC demo/testing
• Can improve accuracy later with real evals
• Validates the vector DB approach before investing in compute

Then, if accuracy is critical:

• Run Option B evaluations for top 100 hardest questions
• Use those to calibrate the estimates
• Best of both worlds: fast MVP + validated accuracy

---

What's your call? Option A to ship today, or Option B for perfect accuracy?