dev/dev_260104_10_config_based_llm_selection.md

[dev_260104_10] LLM Provider Debugging - Config-Based Selection

Date: 2026-01-04 Type: Feature Status: Resolved Stage: [Stage 5: Performance Optimization]

Problem Description

Hard to debug which LLM provider is handling each step with 4-tier fallback chain. Cannot isolate provider performance for improvement.

---

Key Decisions

• Env config: Add LLM_PROVIDER and ENABLE_LLM_FALLBACK to .env
• Routing function: Create _call_with_fallback() to centralize provider selection logic
• Provider mapping: _get_provider_function() maps function names to implementations
• Clear logging: Info logs show exactly which provider is used
• Fallback control: ENABLE_LLM_FALLBACK=false for isolated testing

---

Outcome

Easy debugging: change LLM_PROVIDER in .env or UI to test specific provider. Clear logs show which LLM handled each step.

Deliverables:

• .env - Added LLM_PROVIDER and ENABLE_LLM_FALLBACK config
• src/agent/llm_client.py - Added config-based selection with routing function

Changelog

What was changed:

• .env (~5 lines added)

  • Added LLM_PROVIDER=gemini - Select single provider: "gemini", "huggingface", "groq", or "claude"
  • Added ENABLE_LLM_FALLBACK=false - Toggle fallback behavior (true/false)
  • Removed deprecated DEFAULT_LLM_MODEL config

• src/agent/llm_client.py (~150 lines added/modified)

  • Added LLM_PROVIDER config variable (line 49) - Reads from environment
  • Added ENABLE_LLM_FALLBACK config variable (line 50) - Reads from environment
  • Added _get_provider_function() helper (lines 114-158) - Maps function names to provider implementations
  • Added _call_with_fallback() routing function (lines 161-212)
    • Primary provider: Uses LLM_PROVIDER config
    • Fallback behavior: Controlled by ENABLE_LLM_FALLBACK
    • Logging: Clear info logs showing which provider is used
    • Error handling: Specific error messages when fallback disabled
  • Updated plan_question() - Now uses _call_with_fallback() (simplified from 40 lines to 1 line)
  • Updated select_tools_with_function_calling() - Now uses _call_with_fallback() (simplified from 40 lines to 1 line)
  • Updated synthesize_answer() - Now uses _call_with_fallback() (simplified from 40 lines to 1 line)