mangubee Claude commited on
Commit
4496081
Β·
1 Parent(s): b68b317

debug: log full transcript content for debugging

Browse files

Add comprehensive logging to see exact LLM context during synthesis:
- Log full evidence content in answer_node (before synthesis)
- Log full system + user prompts in synthesize_answer_hf
- Helps debug why LLM says "Unable to answer" despite having transcript

Co-Authored-By: Claude <noreply@anthropic.com>

Files changed (3) hide show
  1. WORKSPACE.md +67 -0
  2. src/agent/graph.py +17 -1
  3. src/agent/llm_client.py +14 -0
WORKSPACE.md ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-01-13 02:26:34,425 - src.agent.graph - INFO - [plan_node] ========== PLAN NODE START ==========
2
+ 2026-01-13 02:26:34,426 - src.agent.graph - INFO - [plan_node] Question: In the video https://www.youtube.com/watch?v=L1vXCYZAYYM, what is the highest number of bird species to be on camera simultaneously?
3
+ 2026-01-13 02:26:34,426 - src.agent.graph - INFO - [plan_node] File paths: None
4
+ 2026-01-13 02:26:34,426 - src.agent.graph - INFO - [plan_node] Available tools: ['web_search', 'parse_file', 'calculator', 'vision', 'youtube_transcript', 'transcribe_audio']
5
+ 2026-01-13 02:26:34,427 - src.agent.graph - INFO - [plan_node] Calling plan_question() with LLM...
6
+ 2026-01-13 02:26:34,428 - src.agent.llm_client - INFO - [plan_question] Using provider: huggingface
7
+ 2026-01-13 02:26:34,428 - src.agent.llm_client - INFO - Initializing HuggingFace Inference client with model: Qwen/Qwen2.5-72B-Instruct
8
+ 2026-01-13 02:26:34,428 - src.agent.llm_client - INFO - [plan_question_hf] Calling HuggingFace (Qwen/Qwen2.5-72B-Instruct) for planning
9
+ GAIAAgent processing question (first 50 chars): In the video https://www.youtube.com/watch?v=L1vXC...
10
+ 2026-01-13 02:26:42,895 - httpx - INFO - HTTP Request: POST https://router.huggingface.co/v1/chat/completions "HTTP/1.1 200 OK"
11
+ 2026-01-13 02:26:42,898 - src.agent.llm_client - INFO - [plan_question_hf] Generated plan (660 chars)
12
+ 2026-01-13 02:26:42,898 - src.agent.graph - INFO - [plan_node] βœ“ Plan created successfully (660 chars)
13
+ 2026-01-13 02:26:42,899 - src.agent.graph - INFO - [plan_node] ========== PLAN NODE END ==========
14
+ 2026-01-13 02:26:42,900 - src.agent.graph - INFO - [execute_node] ========== EXECUTE NODE START ==========
15
+ 2026-01-13 02:26:42,900 - src.agent.graph - INFO - [execute_node] Plan: 1. Use the `youtube_transcript` tool to extract the transcript from the provided YouTube video URL (https://www.youtube.com/watch?v=L1vXCYZAYYM). 2. Review the extracted transcript to identify any mentions of the number of bird species seen simultaneously in the video. 3. If the transcript does not provide the specific information, use the `web_search` tool to search for any reviews, summaries, or analyses of the video that might mention the highest number of bird species seen at once. 4. If the information is still not available, note that the question cannot be answered with the available tools and may require direct observation of the video content.
16
+ 2026-01-13 02:26:42,900 - src.agent.graph - INFO - [execute_node] Question: In the video https://www.youtube.com/watch?v=L1vXCYZAYYM, what is the highest number of bird species to be on camera simultaneously?
17
+ 2026-01-13 02:26:42,901 - src.agent.graph - INFO - [execute_node] Calling select_tools_with_function_calling()...
18
+ 2026-01-13 02:26:42,901 - src.agent.llm_client - INFO - [select_tools] Using provider: huggingface
19
+ 2026-01-13 02:26:42,902 - src.agent.llm_client - INFO - Initializing HuggingFace Inference client with model: Qwen/Qwen2.5-72B-Instruct
20
+ 2026-01-13 02:26:42,902 - src.agent.llm_client - INFO - [select_tools_hf] Calling HuggingFace with function calling for 6 tools, file_paths=None
21
+ 2026-01-13 02:26:44,769 - httpx - INFO - HTTP Request: POST https://router.huggingface.co/v1/chat/completions "HTTP/1.1 200 OK"
22
+ 2026-01-13 02:26:44,771 - src.agent.llm_client - INFO - [select_tools_hf] HuggingFace selected 1 tool(s)
23
+ 2026-01-13 02:26:44,771 - src.agent.graph - INFO - [execute_node] βœ“ LLM selected 1 tool(s)
24
+ 2026-01-13 02:26:44,772 - src.agent.graph - INFO - [execute_node] --- Tool 1/1: youtube_transcript ---
25
+ 2026-01-13 02:26:44,772 - src.agent.graph - INFO - [execute_node] Parameters: {'url': 'https://www.youtube.com/watch?v=L1vXCYZAYYM'}
26
+ 2026-01-13 02:26:44,773 - src.agent.graph - INFO - [execute_node] Executing youtube_transcript...
27
+ 2026-01-13 02:26:44,774 - src.tools.youtube - INFO - Processing YouTube video: L1vXCYZAYYM
28
+ 2026-01-13 02:26:44,784 - src.tools.youtube - INFO - Fetching transcript for video: L1vXCYZAYYM
29
+ 2026-01-13 02:26:45,466 - src.tools.youtube - ERROR - YouTube transcript API failed:
30
+ Could not retrieve a transcript for the video https://www.youtube.com/watch?v=L1vXCYZAYYM! This is most likely caused by:
31
+
32
+ Subtitles are disabled for this video
33
+
34
+ If you are sure that the described cause is not responsible for this error and that a transcript should be retrievable, please create an issue at https://github.com/jdepoix/youtube-transcript-api/issues. Please add which version of youtube_transcript_api you are using and provide the information needed to replicate the error. Also make sure that there are no open issues which already describe your problem!
35
+ 2026-01-13 02:26:45,470 - src.tools.youtube - INFO - Transcript API failed, trying audio transcription...
36
+ 2026-01-13 02:26:45,525 - src.tools.youtube - INFO - Downloading audio from: https://www.youtube.com/watch?v=L1vXCYZAYYM
37
+
38
+ 2026-01-13 02:26:48,123 - src.tools.youtube - INFO - Audio downloaded: /var/folders/05/8vqqybgj751**dmlh3w536dh0000gn/T/youtube_audio_28749.mp3 (1930412 bytes)
39
+ 2026-01-13 02:26:48,123 - src.tools.audio - INFO - Transcribing audio: /var/folders/05/8vqqybgj751**dmlh3w536dh0000gn/T/youtube_audio_28749.mp3
40
+ 2026-01-13 02:26:48,354 - src.tools.audio - INFO - Loading Whisper model: small
41
+ 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 461M/461M [00:07<00:00, 67.9MiB/s]
42
+ 2026-01-13 02:26:57,343 - src.tools.audio - INFO - Whisper model loaded on cpu
43
+ 2026-01-13 02:27:04,275 - src.tools.audio - INFO - Transcription successful: 738 characters
44
+ 2026-01-13 02:27:04,276 - src.tools.youtube - INFO - Cleaned up temp file: /var/folders/05/8vqqybgj751**dmlh3w536dh0000gn/T/youtube_audio_28749.mp3
45
+ 2026-01-13 02:27:04,276 - src.tools.youtube - INFO - Transcript retrieved via Whisper: 738 characters
46
+ 2026-01-13 02:27:04,276 - src.agent.graph - INFO - [execute_node] βœ“ youtube_transcript completed successfully
47
+ 2026-01-13 02:27:04,277 - src.agent.graph - INFO - [execute_node] Summary: 1 tool(s) executed, 1 evidence items collected
48
+ 2026-01-13 02:27:04,277 - src.agent.graph - INFO - [execute_node] ========== EXECUTE NODE END ==========
49
+ 2026-01-13 02:27:04,277 - src.agent.graph - INFO - [answer_node] ========== ANSWER NODE START ==========
50
+ 2026-01-13 02:27:04,278 - src.agent.graph - INFO - [answer_node] Evidence items collected: 1
51
+ 2026-01-13 02:27:04,278 - src.agent.graph - INFO - [answer_node] Errors accumulated: 0
52
+ 2026-01-13 02:27:04,278 - src.agent.graph - INFO - [answer_node] Calling synthesize_answer() with 1 evidence items...
53
+ 2026-01-13 02:27:04,278 - src.agent.llm_client - INFO - [synthesize_answer] Using provider: huggingface
54
+ 2026-01-13 02:27:04,278 - src.agent.llm_client - INFO - Initializing HuggingFace Inference client with model: Qwen/Qwen2.5-72B-Instruct
55
+ 2026-01-13 02:27:04,278 - src.agent.llm_client - INFO - [synthesize_answer_hf] Calling HuggingFace for answer synthesis
56
+ 2026-01-13 02:27:05,281 - httpx - INFO - HTTP Request: POST https://router.huggingface.co/v1/chat/completions "HTTP/1.1 200 OK"
57
+ 2026-01-13 02:27:05,283 - src.agent.llm_client - INFO - [synthesize_answer_hf] Generated answer: Unable to answer
58
+ 2026-01-13 02:27:05,284 - src.agent.graph - INFO - [answer_node] βœ“ Answer generated successfully: Unable to answer
59
+ 2026-01-13 02:27:05,284 - src.agent.graph - INFO - [answer_node] ========== ANSWER NODE END ==========
60
+ 2026-01-13 02:27:05,285 - **main** - INFO - [1/1] Completed a1e91b78
61
+ 2026-01-13 02:27:05,286 - **main** - INFO - Progress: 1/1 questions processed
62
+ GAIAAgent returning answer: Unable to answer
63
+ Agent finished. Submitting 1 answers for user 'mangubee'...
64
+ Submitting 1 answers to: https://agents-course-unit4-scoring.hf.space/submit
65
+ 2026-01-13 02:27:06,261 - **main** - INFO - Total execution time: 34.76 seconds (0m 34s)
66
+ 2026-01-13 02:27:06,266 - **main\_\_ - INFO - Results exported to: /Users/mangubee/Documents/Python/16_HuggingFace/Final_Assignment_Template/\_cache/gaia_results_20260113_022706.json
67
+ Submission successful.
src/agent/graph.py CHANGED
@@ -483,8 +483,24 @@ def answer_node(state: AgentState) -> AgentState:
483
  """
484
  logger.info(f"[answer_node] ========== ANSWER NODE START ==========")
485
  logger.info(f"[answer_node] Evidence items collected: {len(state['evidence'])}")
486
- logger.debug(f"[answer_node] Evidence: {state['evidence']}")
487
  logger.info(f"[answer_node] Errors accumulated: {len(state['errors'])}")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
488
  if state["errors"]:
489
  logger.warning(f"[answer_node] Error list: {state['errors']}")
490
 
 
483
  """
484
  logger.info(f"[answer_node] ========== ANSWER NODE START ==========")
485
  logger.info(f"[answer_node] Evidence items collected: {len(state['evidence'])}")
 
486
  logger.info(f"[answer_node] Errors accumulated: {len(state['errors'])}")
487
+
488
+ # ============================================================================
489
+ # FULL EVIDENCE LOGGING - Debug what evidence is being passed to synthesis
490
+ # ============================================================================
491
+ logger.info("=" * 80)
492
+ logger.info("[EVIDENCE] Full evidence content being passed to synthesis:")
493
+ logger.info("=" * 80)
494
+ for i, ev in enumerate(state['evidence']):
495
+ logger.info(f"[EVIDENCE {i+1}/{len(state['evidence'])}]")
496
+ logger.info(f"{ev[:500]}..." if len(ev) > 500 else f"{ev}")
497
+ logger.info("-" * 80)
498
+ logger.info("=" * 80)
499
+ logger.info("[EVIDENCE] End of evidence content")
500
+ logger.info("=" * 80)
501
+ # ============================================================================
502
+
503
+ logger.debug(f"[answer_node] Evidence: {state['evidence']}")
504
  if state["errors"]:
505
  logger.warning(f"[answer_node] Error list: {state['errors']}")
506
 
src/agent/llm_client.py CHANGED
@@ -1114,6 +1114,20 @@ Extract the factoid answer from the evidence above. Return only the factoid, not
1114
  {"role": "user", "content": user_prompt},
1115
  ]
1116
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1117
  response = client.chat_completion(
1118
  messages=messages,
1119
  max_tokens=256, # Factoid answers are short
 
1114
  {"role": "user", "content": user_prompt},
1115
  ]
1116
 
1117
+ # ============================================================================
1118
+ # FULL CONTEXT LOGGING - Debug LLM synthesis failures
1119
+ # ============================================================================
1120
+ logger.info("=" * 80)
1121
+ logger.info("[LLM CONTEXT] Full synthesis prompt being sent to LLM:")
1122
+ logger.info("=" * 80)
1123
+ logger.info(f"[SYSTEM PROMPT]\n{system_prompt}")
1124
+ logger.info("-" * 80)
1125
+ logger.info(f"[USER PROMPT]\n{user_prompt}")
1126
+ logger.info("=" * 80)
1127
+ logger.info("[LLM CONTEXT] End of full context")
1128
+ logger.info("=" * 80)
1129
+ # ============================================================================
1130
+
1131
  response = client.chat_completion(
1132
  messages=messages,
1133
  max_tokens=256, # Factoid answers are short