Commit History
-
fix: correct author name formatting in multiple files
e7b4937
-
Enhance YouTube video processing with transcript and frame analysis modesβ
- Updated logging standard to use Markdown format for session logs. - Modified `run_and_submit_all` to include a new `video_mode` parameter for selecting YouTube processing mode (Transcript or Frames). - Removed obsolete brainstorming document for YouTube transcript support. - Added OpenCV and other dependencies for frame extraction in `pyproject.toml` and `requirements.txt`. - Refactored `llm_client.py` to log session details in Markdown format. - Implemented `youtube.py` to support both transcript extraction and frame analysis, with appropriate logging and error handling. - Updated tool descriptions to reflect new functionality for analyzing video frames. - Added backward compatibility for the `youtube_transcript` function to respect the `YOUTUBE_MODE` environment variable.f1b095a
-
refactor: rename runtime folders with underscore prefixβ
Rename folders to follow Python convention for internal/private: - log/ β _log/ (runtime logs, debugging) - output/ β _output/ (runtime results, user downloads) - input/ β _input/ (user testing files, not app input) Rationale: _ prefix signals "internal, temporary, not part of public API" Updated files: - src/agent/llm_client.py - Path("log") β Path("_log") - src/tools/youtube.py - Path("log") β Path("_log") - test/test_phase0_hf_vision_api.py - Path("output") β Path("_output") - .gitignore - Updated folder references (log/, input/ β _log/, _input/, added _output/) - CHANGELOG.md - Added infrastructure entry Co-Authored-By: Claude <noreply@anthropic.com>04ece4c -
feat: consolidate LLM logs to single session fileβ
Implement session-level log file to avoid polluting log/ folder with 20+ files per evaluation. Changes: - Added session log management (get_session_log_file, reset_session_log) - Changed from per-question logs to per-session logs - New format: log/llm_session_YYYYMMDD_HHMMSS.txt - All questions append to single file with QUESTION START/END markers Also added milestone entry: 30% target achieved! Phase 1 (YouTube + Audio) fixed 4 questions. Co-Authored-By: Claude <noreply@anthropic.com>a0fa418 -
feat: add Chain of Thought for LLM synthesis debuggingβ
Implement CoT format to expose LLM reasoning process for debugging synthesis failures. Changes: - Updated system_prompt for all 3 providers (HF, Groq, Claude) to request REASONING + FINAL ANSWER format - Increased max_tokens from 256 to 1024 to accommodate reasoning - Added parsing logic to extract FINAL ANSWER from response - Enhanced log file format to save full response with reasoning - Updated CHANGELOG.md Result: Can now inspect LLM's thought process in log/llm_context_*.txt files Co-Authored-By: Claude <noreply@anthropic.com>c64957b -
fix: close parenthesis in execute_node error handlingβ
Co-Authored-By: Claude <noreply@anthropic.com>9c1dff0 -
feat: standardize logging with console + file separationβ
- Console: Compressed status workflow ([plan] β, [execute] β, [answer] β) - Files: log/ folder with full LLM context and transcripts - Update CLAUDE.md with logging standard - Add log/ to .gitignore Separates status updates (console) from debugging details (log/ files). Co-Authored-By: Claude <noreply@anthropic.com>fa82033 -
refactor: compress console logs for clarityβ
Console logs now show status only: - [plan] β 660 chars - [execute] 1 tool(s) selected - [1/1] youtube_transcript β - [execute] 1 tools, 1 evidence - [answer] β 3 Full context saved to _cache/llm_context_*.txt for debugging. Co-Authored-By: Claude <noreply@anthropic.com>fed2514 -
feat: export LLM synthesis context to cache for debuggingβ
Save full LLM context to _cache/llm_context_TIMESTAMP.txt: - System prompt - User prompt with evidence - Raw evidence items - LLM's final answer Helps debug why LLM fails despite having correct evidence. Co-Authored-By: Claude <noreply@anthropic.com>86bc1db -
debug: log full transcript content for debuggingβ
Add comprehensive logging to see exact LLM context during synthesis: - Log full evidence content in answer_node (before synthesis) - Log full system + user prompts in synthesize_answer_hf - Helps debug why LLM says "Unable to answer" despite having transcript Co-Authored-By: Claude <noreply@anthropic.com>4496081 -
feat: add transcript caching and upgrade synthesis modelβ
- Add save_transcript_to_cache() to save transcripts for debugging - Upgrade LLM: Qwen 2.5 β openai/gpt-oss-120b (Scaleway) - Document HF provider suffix behavior (auto-routing is bad practice) - Model iteration: Qwen 2.5 β Llama 3.3 β gpt-oss-120b Co-Authored-By: Claude <noreply@anthropic.com>b68b317 -
fix: add youtube_transcript and transcribe_audio to TOOL_FUNCTIONSβ
Bug: Tools were registered in TOOLS dict but missing from TOOL_FUNCTIONS mapping in graph.py, causing "Tool 'youtube_transcript' not found in TOOL_FUNCTIONS" error. Fixed: - Added imports: youtube_transcript, transcribe_audio - Updated TOOL_FUNCTIONS dict in execute_node (line 268) - Updated TOOL_FUNCTIONS dict in fallback section (line 410) Co-Authored-By: Claude <noreply@anthropic.com>40db96d
-
feat: system error field, calculator fix, target task IDs, course vs GAIA docsβ
Changes: - System error field: Changed to boolean yes/no with error_log - Calculator threading fix: Handle signal.alarm() failure in threads - Target task IDs: Debug feature to run specific questions - Course vs GAIA: Documentation distinguishing course API from official GAIA - Quick test script: test/test_quick_fixes.py for targeted testing Modified: - app.py: System error field, target task IDs UI, submission logic - src/tools/calculator.py: Thread-safe timeout handling - src/agent/graph.py: Evidence formatting for dict results - src/agent/llm_client.py: Fallback mechanism archived - CHANGELOG.md: All changes documented - README.md: Added submission guide reference Added: - docs/gaia_submission_guide.md: Complete submission guide - test/test_quick_fixes.py: Targeted question testing Co-Authored-By: Claude <noreply@anthropic.com>41ac444
-
Update Groq model to openai/gpt-oss-120b
44d1862 -
Fix: Read LLM provider config at runtime for UI updates
91f84b2 -
Add UI-based LLM provider selection for cloud testingβ
- Config-based LLM selection: Added LLM_PROVIDER and ENABLE_LLM_FALLBACK environment variables - Smart routing: _call_with_fallback() function routes to selected provider with optional fallback - UI dropdowns: Added LLM provider selection to both Test & Debug and Full Evaluation tabs - Cloud-friendly: UI selection overrides env vars, enabling instant provider switching without rebuilds - Unified behavior: Same UI selection works identically in local and cloud environments Modified files: - src/agent/llm_client.py: Config-based provider routing (~150 lines) - app.py: UI dropdowns and function parameter updates (~30 lines) - CHANGELOG.md: Documented two problems solved π€ Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>ac73681 -
Groq Model Update
6e73cae -
Stage 5: Performance optimization - retry logic, Groq integration, improved promptsβ
- Added exponential backoff retry logic (3 attempts, 1s/2s/4s delays) - Integrated Groq as 4th free LLM tier (Llama 3.1 70B) - Improved tool selection prompts with few-shot examples - Added graceful vision question skip logic - Relaxed calculator validation (graceful errors) - Improved TOOLS schema descriptions - All 99 tests passing π€ Generated with Claude Code Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>5890f66
-
Fix: Tool name consistency across TOOLS registry and TOOL_FUNCTIONSβ
Resolved mismatch between: - TOOLS registry (src/tools/__init__.py): "web_search", "calculator", "vision" - TOOL_FUNCTIONS (src/agent/graph.py): Was using "search", "safe_eval", "analyze_image" Changes: - Updated TOOL_FUNCTIONS to match TOOLS registry names - Updated fallback_tool_selection() to use correct names - Fixed test mock to use "web_search" instead of "search" Result: HuggingFace LLM function calling now works correctly Tests: 99/99 passing β π€ Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>59f3cce -
Feat: Add HuggingFace Inference API as free LLM fallback tierβ
Stage 4 completion - Added 3-tier fallback architecture: - Tier 1: Gemini 2.0 Flash (free, daily quota) - Tier 2: HuggingFace Qwen 2.5 72B (free, rate limited) - NEW - Tier 3: Claude Sonnet 4.5 (paid) - Tier 4: Keyword matching (deterministic) Changes: - Added HF integration to llm_client.py (~150 lines) - Added HF_TOKEN validation in graph.py - Updated UI to show HF_TOKEN status in app.py - Fixed TOOLS schema bug (list β dict format) - Created comprehensive dev log Tests: 99/99 passing β π€ Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>4eed151
-
Fix: Make LLM provider configurable (Gemini primary, Claude fallback)β
**Consistency Fix:** Stage 2 tools used Gemini primary + Claude fallback pattern. Stage 3 now matches this pattern for all LLM operations. **Multi-Provider Implementation:** - Gemini 2.0 Flash (primary, free tier, 1500 req/day) - Claude Sonnet 4.5 (fallback, paid, highest quality) - All 3 LLM functions support both providers with automatic fallback **New Functions:** - plan_question_gemini() + plan_question_claude() β plan_question() - select_tools_gemini() + select_tools_claude() β select_tools_with_function_calling() - synthesize_answer_gemini() + synthesize_answer_claude() β synthesize_answer() **Pattern:** Try Gemini first, fallback to Claude on any error. Same pattern as vision tool (analyze_image_gemini β analyze_image_claude). **Dependencies:** - Added google-generativeai package **Testing:** - All 99 tests passing - Tests use mocks, work with both providers π€ Generated with Claude Code Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>bbc52c6 -
Stage 3: Core Logic Implementation - LLM Integrationβ
Implemented Stage 3 core agent logic with full LLM integration: **New Features:** - LLM-based planning: Analyzes questions and generates execution plans - Dynamic tool selection: Claude function calling for tool selection - Parameter extraction: LLM extracts tool parameters from questions - Answer synthesis: LLM generates factoid answers from evidence - Conflict resolution: LLM evaluates contradictory information **New Files:** - src/agent/llm_client.py - Centralized LLM client - test/test_llm_integration.py - 8 new LLM integration tests - test/test_stage3_e2e.py - Manual E2E test script **Modified Files:** - src/agent/graph.py - Implemented all node logic (plan/execute/answer) - AgentState schema: Added file_paths, tool_results, evidence fields **Framework Updates:** - Updated dev records (04-07) with new framework parameters - Added missing parameters from framework v2026-01-02 **Test Results:** - All 99 tests passing (6 Stage 1 + 85 Stage 2 + 8 Stage 3) - No regressions from prβ¦4c0b7eb -
Stage 2: Implement tool development with retry logic and error handlingβ
Implemented 4 core tools with comprehensive test coverage: **Tools Added:** - web_search.py: Tavily/Exa search with fallback (10 tests) - file_parser.py: PDF/Excel/Word/Text parsing (19 tests) - calculator.py: Safe math eval with security (41 tests) - vision.py: Multimodal image analysis (15 tests) **Features:** - Retry logic with tenacity (exponential backoff, 3 max retries) - Comprehensive error handling and logging - Tool registry in __init__.py with metadata - 85 passing tests total **Integration:** - Updated graph.py execute_node to load tool registry - Added TOOLS dict for Stage 3 dynamic tool selection - Maintained Stage 1 compatibility **Testing:** - Created test fixtures for all file types - Mock API testing for web search and vision - Security testing for calculator (prevents code injection) - All 91 tests passing (6 agent + 85 tool tests) π€ Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>1041734
-
Stage 1: Foundation Setup - LangGraph agent with isolated environmentβ
- Implemented 3-node StateGraph (plan β execute β answer) - Added isolated uv environment (pyproject.toml, 102 packages) - Configured Tavily (free tier) as default search tool - Security: .env.example template, .gitignore protection - Tests: Unit tests + integration verification passing - Ready for Stage 2: Tool development π€ Generated with Claude Code Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>bd73133