Commit History
-
fix: correct author name formatting in multiple files
e7b4937
-
Enhance YouTube video processing with transcript and frame analysis modesβ
- Updated logging standard to use Markdown format for session logs. - Modified `run_and_submit_all` to include a new `video_mode` parameter for selecting YouTube processing mode (Transcript or Frames). - Removed obsolete brainstorming document for YouTube transcript support. - Added OpenCV and other dependencies for frame extraction in `pyproject.toml` and `requirements.txt`. - Refactored `llm_client.py` to log session details in Markdown format. - Implemented `youtube.py` to support both transcript extraction and frame analysis, with appropriate logging and error handling. - Updated tool descriptions to reflect new functionality for analyzing video frames. - Added backward compatibility for the `youtube_transcript` function to respect the `YOUTUBE_MODE` environment variable.f1b095a
-
refactor: rename runtime folders with underscore prefixβ
Rename folders to follow Python convention for internal/private: - log/ β _log/ (runtime logs, debugging) - output/ β _output/ (runtime results, user downloads) - input/ β _input/ (user testing files, not app input) Rationale: _ prefix signals "internal, temporary, not part of public API" Updated files: - src/agent/llm_client.py - Path("log") β Path("_log") - src/tools/youtube.py - Path("log") β Path("_log") - test/test_phase0_hf_vision_api.py - Path("output") β Path("_output") - .gitignore - Updated folder references (log/, input/ β _log/, _input/, added _output/) - CHANGELOG.md - Added infrastructure entry Co-Authored-By: Claude <noreply@anthropic.com>04ece4c -
feat: consolidate LLM logs to single session fileβ
Implement session-level log file to avoid polluting log/ folder with 20+ files per evaluation. Changes: - Added session log management (get_session_log_file, reset_session_log) - Changed from per-question logs to per-session logs - New format: log/llm_session_YYYYMMDD_HHMMSS.txt - All questions append to single file with QUESTION START/END markers Also added milestone entry: 30% target achieved! Phase 1 (YouTube + Audio) fixed 4 questions. Co-Authored-By: Claude <noreply@anthropic.com>a0fa418 -
feat: add Chain of Thought for LLM synthesis debuggingβ
Implement CoT format to expose LLM reasoning process for debugging synthesis failures. Changes: - Updated system_prompt for all 3 providers (HF, Groq, Claude) to request REASONING + FINAL ANSWER format - Increased max_tokens from 256 to 1024 to accommodate reasoning - Added parsing logic to extract FINAL ANSWER from response - Enhanced log file format to save full response with reasoning - Updated CHANGELOG.md Result: Can now inspect LLM's thought process in log/llm_context_*.txt files Co-Authored-By: Claude <noreply@anthropic.com>c64957b -
fix: close parenthesis in execute_node error handlingβ
Co-Authored-By: Claude <noreply@anthropic.com>9c1dff0 -
feat: standardize logging with console + file separationβ
- Console: Compressed status workflow ([plan] β, [execute] β, [answer] β) - Files: log/ folder with full LLM context and transcripts - Update CLAUDE.md with logging standard - Add log/ to .gitignore Separates status updates (console) from debugging details (log/ files). Co-Authored-By: Claude <noreply@anthropic.com>fa82033 -
refactor: compress console logs for clarityβ
Console logs now show status only: - [plan] β 660 chars - [execute] 1 tool(s) selected - [1/1] youtube_transcript β - [execute] 1 tools, 1 evidence - [answer] β 3 Full context saved to _cache/llm_context_*.txt for debugging. Co-Authored-By: Claude <noreply@anthropic.com>fed2514 -
feat: export LLM synthesis context to cache for debuggingβ
Save full LLM context to _cache/llm_context_TIMESTAMP.txt: - System prompt - User prompt with evidence - Raw evidence items - LLM's final answer Helps debug why LLM fails despite having correct evidence. Co-Authored-By: Claude <noreply@anthropic.com>86bc1db -
debug: log full transcript content for debuggingβ
Add comprehensive logging to see exact LLM context during synthesis: - Log full evidence content in answer_node (before synthesis) - Log full system + user prompts in synthesize_answer_hf - Helps debug why LLM says "Unable to answer" despite having transcript Co-Authored-By: Claude <noreply@anthropic.com>4496081 -
feat: add transcript caching and upgrade synthesis modelβ
- Add save_transcript_to_cache() to save transcripts for debugging - Upgrade LLM: Qwen 2.5 β openai/gpt-oss-120b (Scaleway) - Document HF provider suffix behavior (auto-routing is bad practice) - Model iteration: Qwen 2.5 β Llama 3.3 β gpt-oss-120b Co-Authored-By: Claude <noreply@anthropic.com>b68b317 -
debug: log full transcript content for debuggingβ
Added logging of full transcript text to inspect why AI fails to extract answer. Logs will show first 200 chars for API, full transcript for Whisper fallback. Co-Authored-By: Claude <noreply@anthropic.com>a1d5de5 -
fix: vision tool cannot process videos, clarify image-only capabilityβ
Removed misleading statement about video files. Vision tool ONLY processes image files (JPEG, PNG, GIF, BMP, WebP), not videos. Changes: - Changed "Analyze images or videos" to "Analyze image files" - Added explicit image format list (JPEG, PNG, GIF, BMP, WebP) - Removed "For video files...file must be downloaded first" - Added "Cannot process videos" NOTE - Added guidance to use youtube_transcript/transcribe_audio for videos Co-Authored-By: Claude <noreply@anthropic.com>3ee1e2c -
fix: clarify tool descriptions to prevent LLM selecting vision for YouTubeβ
Problem: LLM was selecting vision tool for YouTube URLs, which cannot process them directly because vision description mentioned "YouTube links". Changes: - Vision: Removed "videos, YouTube links" from description - Vision: Added NOTE about using youtube_transcript for YouTube - YouTube: Enhanced description to be more explicit about being FIRST choice - YouTube: Added "ONLY tool that can process YouTube URLs directly" This should guide LLM to select youtube_transcript for YouTube URLs. Co-Authored-By: Claude <noreply@anthropic.com>61ecfdb -
fix: typo WISPER_MODEL -> WHISPER_MODEL in audio.pyβ
Typo caused NameError during Whisper transcription. Fixed by correcting variable name reference. Co-Authored-By: Claude <noreply@anthropic.com>7bb1d7c -
fix: YouTubeTranscriptApi.fetch is instance method not class methodβ
The API changed: fetch() requires an instance. Added: api = YouTubeTranscriptApi() before calling api.fetch() Co-Authored-By: Claude <noreply@anthropic.com>92346a5 -
fix: youtube transcript API and missing datasets dependencyβ
- Changed YouTubeTranscriptApi.get_transcript to .fetch (API changed) - Added datasets>=4.4.0 to pyproject.toml (was missing) Co-Authored-By: Claude <noreply@anthropic.com>a170083 -
fix: add youtube_transcript and transcribe_audio to TOOL_FUNCTIONSβ
Bug: Tools were registered in TOOLS dict but missing from TOOL_FUNCTIONS mapping in graph.py, causing "Tool 'youtube_transcript' not found in TOOL_FUNCTIONS" error. Fixed: - Added imports: youtube_transcript, transcribe_audio - Updated TOOL_FUNCTIONS dict in execute_node (line 268) - Updated TOOL_FUNCTIONS dict in fallback section (line 410) Co-Authored-By: Claude <noreply@anthropic.com>40db96d -
feat: Phase 1 - YouTube transcript + Whisper audio transcriptionβ
Implementation complete for Phase 1 (YouTube support): New tools: - src/tools/audio.py: Whisper transcription with @spaces.GPU decorator * ZeroGPU acceleration (5-10x speedup vs CPU) * Supports MP3, WAV, M4A, OGG, FLAC, AAC * Model caching for efficient repeated use * Unified tool for Phase 1 (YouTube fallback) and Phase 2 (MP3 files) - src/tools/youtube.py: YouTube transcript extraction with audio fallback * Primary: youtube-transcript-api (instant, 1-3 seconds) * Fallback: yt-dlp audio extraction + Whisper (30s-2min) * Handles youtube.com, youtu.be, shorts URLs * Returns clean transcript for LLM analysis Updated files: - requirements.txt: Added youtube-transcript-api, openai-whisper, yt-dlp - src/tools/__init__.py: Registered youtube_transcript and transcribe_audio tools - brainstorming_phase1_youtube.md: Documented ZeroGPU requirement, validation results Architecture: YouTube URL β youtube-transcript-api (fast) β fallback: yt-dlp + Whisper (slow) Expected impactβ¦38cc8e4
-
feat: system error field, calculator fix, target task IDs, course vs GAIA docsβ
Changes: - System error field: Changed to boolean yes/no with error_log - Calculator threading fix: Handle signal.alarm() failure in threads - Target task IDs: Debug feature to run specific questions - Course vs GAIA: Documentation distinguishing course API from official GAIA - Quick test script: test/test_quick_fixes.py for targeted testing Modified: - app.py: System error field, target task IDs UI, submission logic - src/tools/calculator.py: Thread-safe timeout handling - src/agent/graph.py: Evidence formatting for dict results - src/agent/llm_client.py: Fallback mechanism archived - CHANGELOG.md: All changes documented - README.md: Added submission guide reference Added: - docs/gaia_submission_guide.md: Complete submission guide - test/test_quick_fixes.py: Targeted question testing Co-Authored-By: Claude <noreply@anthropic.com>41ac444 -
Phase 0-2: HF Vision Integration Complete - Google Gemma 3 Selectedβ
Phase 0 (Extended Testing): - Tested 4 additional user-requested models - Validated: google/gemma-3-27b-it:scaleway (6s, RECOMMENDED) - Validated: zai-org/GLM-4.6V-Flash:zai-org (16s) - Validated: Qwen/Qwen3-VL-30B-A3B-Instruct:novita (14s) - Failed: GLM-4.7, gpt-oss-120b (text-only models) Phase 1 (Implementation): - Added analyze_image_hf() in src/tools/vision.py - Fixed analyze_image() routing to respect LLM_PROVIDER - Added HF_TOKEN, HF_VISION_MODEL to settings - Each provider fails independently (no fallback chains) Phase 2 (Smoke Tests): - Created test/test_smoke_hf_vision.py - Smoke test PASSED: red square image correctly identified - Fixed Settings.hf_token integration - Removed unsupported timeout parameter Modified Files: - src/tools/vision.py: +120 lines (HF vision function + routing fix) - src/config/settings.py: +5 lines (HF config) - .env.example: +7 lines (HF_TOKEN, HF_VISION_MODEL docs) - CHANGELOG.md: +93 lines (Phase 0-2 documentation) - PLAN.md: +20 lines (Phase 0β¦2bb2d3d
-
Update Groq model to openai/gpt-oss-120b
44d1862 -
Add Groundtruth Answer Column
dc583a7 -
Integrate benchmark dataset with results from HF as groundtruth
9fb23b8 -
Fix: Read LLM provider config at runtime for UI updates
91f84b2 -
Add UI-based LLM provider selection for cloud testingβ
- Config-based LLM selection: Added LLM_PROVIDER and ENABLE_LLM_FALLBACK environment variables - Smart routing: _call_with_fallback() function routes to selected provider with optional fallback - UI dropdowns: Added LLM provider selection to both Test & Debug and Full Evaluation tabs - Cloud-friendly: UI selection overrides env vars, enabling instant provider switching without rebuilds - Unified behavior: Same UI selection works identically in local and cloud environments Modified files: - src/agent/llm_client.py: Config-based provider routing (~150 lines) - app.py: UI dropdowns and function parameter updates (~30 lines) - CHANGELOG.md: Documented two problems solved π€ Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>ac73681 -
Groq Model Update
6e73cae -
Stage 5: Performance optimization - retry logic, Groq integration, improved promptsβ
- Added exponential backoff retry logic (3 attempts, 1s/2s/4s delays) - Integrated Groq as 4th free LLM tier (Llama 3.1 70B) - Improved tool selection prompts with few-shot examples - Added graceful vision question skip logic - Relaxed calculator validation (graceful errors) - Improved TOOLS schema descriptions - All 99 tests passing π€ Generated with Claude Code Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>5890f66
-
Fix: Tool name consistency across TOOLS registry and TOOL_FUNCTIONSβ
Resolved mismatch between: - TOOLS registry (src/tools/__init__.py): "web_search", "calculator", "vision" - TOOL_FUNCTIONS (src/agent/graph.py): Was using "search", "safe_eval", "analyze_image" Changes: - Updated TOOL_FUNCTIONS to match TOOLS registry names - Updated fallback_tool_selection() to use correct names - Fixed test mock to use "web_search" instead of "search" Result: HuggingFace LLM function calling now works correctly Tests: 99/99 passing β π€ Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>59f3cce -
Feat: Add HuggingFace Inference API as free LLM fallback tierβ
Stage 4 completion - Added 3-tier fallback architecture: - Tier 1: Gemini 2.0 Flash (free, daily quota) - Tier 2: HuggingFace Qwen 2.5 72B (free, rate limited) - NEW - Tier 3: Claude Sonnet 4.5 (paid) - Tier 4: Keyword matching (deterministic) Changes: - Added HF integration to llm_client.py (~150 lines) - Added HF_TOKEN validation in graph.py - Updated UI to show HF_TOKEN status in app.py - Fixed TOOLS schema bug (list β dict format) - Created comprehensive dev log Tests: 99/99 passing β π€ Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>4eed151
-
Fix: Make LLM provider configurable (Gemini primary, Claude fallback)β
**Consistency Fix:** Stage 2 tools used Gemini primary + Claude fallback pattern. Stage 3 now matches this pattern for all LLM operations. **Multi-Provider Implementation:** - Gemini 2.0 Flash (primary, free tier, 1500 req/day) - Claude Sonnet 4.5 (fallback, paid, highest quality) - All 3 LLM functions support both providers with automatic fallback **New Functions:** - plan_question_gemini() + plan_question_claude() β plan_question() - select_tools_gemini() + select_tools_claude() β select_tools_with_function_calling() - synthesize_answer_gemini() + synthesize_answer_claude() β synthesize_answer() **Pattern:** Try Gemini first, fallback to Claude on any error. Same pattern as vision tool (analyze_image_gemini β analyze_image_claude). **Dependencies:** - Added google-generativeai package **Testing:** - All 99 tests passing - Tests use mocks, work with both providers π€ Generated with Claude Code Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>bbc52c6 -
Stage 3: Core Logic Implementation - LLM Integrationβ
Implemented Stage 3 core agent logic with full LLM integration: **New Features:** - LLM-based planning: Analyzes questions and generates execution plans - Dynamic tool selection: Claude function calling for tool selection - Parameter extraction: LLM extracts tool parameters from questions - Answer synthesis: LLM generates factoid answers from evidence - Conflict resolution: LLM evaluates contradictory information **New Files:** - src/agent/llm_client.py - Centralized LLM client - test/test_llm_integration.py - 8 new LLM integration tests - test/test_stage3_e2e.py - Manual E2E test script **Modified Files:** - src/agent/graph.py - Implemented all node logic (plan/execute/answer) - AgentState schema: Added file_paths, tool_results, evidence fields **Framework Updates:** - Updated dev records (04-07) with new framework parameters - Added missing parameters from framework v2026-01-02 **Test Results:** - All 99 tests passing (6 Stage 1 + 85 Stage 2 + 8 Stage 3) - No regressions from prβ¦4c0b7eb -
Move test files from tests/ to test/ folder per project standardβ
- Moved all test files to standard test/ folder (singular, not plural) - test_web_search.py, test_file_parser.py, test_calculator.py, test_vision.py - Moved fixtures/ directory to test/fixtures/ - Removed tests/ directory - All 91 tests still passing8e48c56 -
Stage 2: Implement tool development with retry logic and error handlingβ
Implemented 4 core tools with comprehensive test coverage: **Tools Added:** - web_search.py: Tavily/Exa search with fallback (10 tests) - file_parser.py: PDF/Excel/Word/Text parsing (19 tests) - calculator.py: Safe math eval with security (41 tests) - vision.py: Multimodal image analysis (15 tests) **Features:** - Retry logic with tenacity (exponential backoff, 3 max retries) - Comprehensive error handling and logging - Tool registry in __init__.py with metadata - 85 passing tests total **Integration:** - Updated graph.py execute_node to load tool registry - Added TOOLS dict for Stage 3 dynamic tool selection - Maintained Stage 1 compatibility **Testing:** - Created test fixtures for all file types - Mock API testing for web search and vision - Security testing for calculator (prevents code injection) - All 91 tests passing (6 agent + 85 tool tests) π€ Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>1041734
-
Stage 1: Foundation Setup - LangGraph agent with isolated environmentβ
- Implemented 3-node StateGraph (plan β execute β answer) - Added isolated uv environment (pyproject.toml, 102 packages) - Configured Tavily (free tier) as default search tool - Security: .env.example template, .gitignore protection - Tests: Unit tests + integration verification passing - Ready for Stage 2: Tool development π€ Generated with Claude Code Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>bd73133