最終修復

.cursor/rules/reviewgate.mdc

@@ -1,131 +0,0 @@
----
-description: 
-globs: 
-alwaysApply: true
----
----
-description: 
-globs: 
-alwaysApply: true
----
-# AI Agent Protocol: Self-Contained Mandatory Interactive Final Review Loop with Auto-Script Creation (Version 3)
-
-## 0. User Advisory & Safety:
-This is an advanced, experimental rule. It instructs the AI Agent to check for, potentially create, and then execute a Python script (`final_review_gate.py`) in your project's root directory to facilitate an interactive review loop. By using this rule, you acknowledge that the AI will attempt file system operations (read, write, execute) as directed. Ensure the AI's execution environment has appropriate permissions if you expect script auto-creation to work, and be aware of the security implications.
-
-## 1. Core Principle & Context:
-Your primary task processing includes a **mandatory final interactive review stage** driven by user input through a specific Python script (`final_review_gate.py`). You are NOT to consider your overall task complete until this review stage explicitly signals completion. This protocol leverages your known capability to run commands in a sub-terminal (displayed within the chat interface) and read their output.
-
-## Phase 0: Ensure `final_review_gate.py` Script Exists
-(This phase is executed ONCE per user request that triggers this overall protocol, or if the script is missing or its content is incorrect.)
-
-1.  **Define Script Details:**
-    * **Script Name:** `final_review_gate.py`
-    * **Target Location:** Directly in the root of the current project/workspace.
-    * **Python Script Content (ensure this exact content is used):**
-        ```python
-        # final_review_gate.py
-        import sys
-        import os
-
-        if __name__ == "__main__":
-            # Try to make stdout unbuffered for more responsive interaction.
-            # This might not work on all platforms or if stdout is not a TTY,
-            # but it's a good practice for this kind of interactive script.
-            try:
-                sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', buffering=1)
-            except Exception:
-                pass # Ignore if unbuffering fails, e.g., in certain environments
-
-            try:
-                sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', buffering=1)
-            except Exception:
-                pass # Ignore
-
-            print("--- FINAL REVIEW GATE ACTIVE ---", flush=True)
-            print("AI has completed its primary actions. Awaiting your review or further sub-prompts.", flush=True)
-            print("Type your sub-prompt, or one of: 'TASK_COMPLETE', 'Done', 'Quit', 'q' to signal completion.", flush=True) # MODIFIED
-            
-            active_session = True
-            while active_session:
-                try:
-                    # Signal that the script is ready for input.
-                    # The AI doesn't need to parse this, but it's good for user visibility.
-                    print("REVIEW_GATE_AWAITING_INPUT:", end="", flush=True) 
-                    
-                    line = sys.stdin.readline()
-                    
-                    if not line:  # EOF
-                        print("--- REVIEW GATE: STDIN CLOSED (EOF), EXITING SCRIPT ---", flush=True)
-                        active_session = False
-                        break
-                    
-                    user_input = line.strip()
-
-                    # Check for exit conditions
-                    if user_input.upper() in ['TASK_COMPLETE', 'DONE', 'QUIT', 'Q']: # MODIFIED: Empty string no longer exits
-                        print(f"--- REVIEW GATE: USER SIGNALED COMPLETION WITH '{user_input.upper()}' ---", flush=True)
-                        active_session = False
-                        break
-                    elif user_input: # If there's any other non-empty input (and not a completion command)
-                        # This is the critical line the AI will "listen" for.
-                        print(f"USER_REVIEW_SUB_PROMPT: {user_input}", flush=True)
-                    # If user_input was empty (and not a completion command),
-                    # the loop simply continues, and "REVIEW_GATE_AWAITING_INPUT:" will be printed again.
-                    
-                except KeyboardInterrupt:
-                    print("--- REVIEW GATE: SESSION INTERRUPTED BY USER (KeyboardInterrupt) ---", flush=True)
-                    active_session = False
-                    break
-                except Exception as e:
-                    print(f"--- REVIEW GATE SCRIPT ERROR: {e} ---", flush=True)
-                    active_session = False
-                    break
-                    
-            print("--- FINAL REVIEW GATE SCRIPT EXITED ---", flush=True)
-        ```
-
-2.  **Ensure Script Existence and Correctness:**
-    a.  Use your file system tools to check if the file `final_review_gate.py` exists in the project root.
-    b.  **If the file `final_review_gate.py` does NOT exist OR if its content does not EXACTLY match the Python script provided in section "Phase 0.1.Python Script Content" (you may need to read the existing file and compare):**
-        i.  You MUST create or overwrite the `final_review_gate.py` file in the project root.
-        ii. The content of this file MUST be *exactly* the Python script content provided above. Ensure all indentation, comments, and lines are preserved perfectly. Use your "create file" or "edit file" tool.
-        iii. After creating/overwriting it, announce: "The `final_review_gate.py` script has been created/updated in the project root with the latest interaction enhancements."
-    c.  **If the file `final_review_gate.py` already exists and its content is correct:** You may skip announcing its creation.
-    d.  If you encounter any errors checking for, reading, or creating/writing the file (e.g., permission issues), you MUST report this to the user. State clearly: "Error managing `final_review_gate.py`: [specific error]. Cannot proceed with interactive review." and then conclude the interaction for the main task as best as you can without this interactive step.
-
-## Phase 1: Primary Task Execution
-1.  Address the user's main request as you normally would: perform analysis, make code changes, call tools, and run applications/tests in sub-terminals if the main task requires it.
-    * **Application Execution Confirmation:** Before attempting to run an entire application (e.g., a web server, a full front-end build, a long-running background process), you MUST ask the user for explicit confirmation. For example: "I am about to run the [application name/command]. Is it okay to proceed, or is it possibly already running?"
-    * This confirmation is NOT required for running individual test scripts, linters, build tools for specific components, or other non-application-level commands necessary for task completion.
-    * If the user indicates an application is already running or denies permission, adapt your strategy accordingly (e.g., focus on tests or specific file executions).
-2.  Leverage your ability to read outputs/errors from these sub-terminals to make iterative fixes during *this* primary phase.
-
-## Phase 2: Mandatory Final Interactive Review Script Execution
-1.  **Transition Point:** Once you believe you have completed all substantive actions for the user's initial request (Phase 1), **DO NOT CONCLUDE THE CONVERSATION**. You MUST now transition to this Final Interactive Review phase.
-2.  **Action:** Execute the `final_review_gate.py` script (which you ensured exists and is correct in Phase 0) from the project root.
-    * Determine the correct Python interpreter command (e.g., `python`, `python3`) available in the execution environment. A robust command attempt would be `python3 ./final_review_gate.py` and if that fails, try `python ./final_review_gate.py`.
-    * **Crucially, the terminal for this script MUST run directly within the chat window, and its output MUST be continuously visible to the user as it is generated.** It must NOT pop open as a separate window or run hidden in the background. You MUST wait for the script (and its terminal interaction) to fully complete and exit before you proceed with any further actions or attempt to conclude the overall task. Your processing will be blocked by this script's execution.
-3.  **Announcement:** Clearly inform the user: "I have completed the primary actions. Now initiating the mandatory Final Interactive Review by running `./final_review_gate.py`. The review terminal is now active **in this chat window, and its output will be displayed live**. Please provide any sub-prompts directly into that script's input area, or type 'TASK_COMPLETE', 'Done', 'Quit', or 'q' when satisfied." (MODIFIED: Removed "or simply press Enter")
-
-## Phase 3: Interactive Review Loop (Monitoring Script Output)
-1.  **Active Monitoring & Display:** Continuously monitor the standard output (stdout) of the launched `final_review_gate.py` script **as it appears live in the chat window**. Ensure the user sees all script output, including the `REVIEW_GATE_AWAITING_INPUT:` prompt. You will "read" its output using the same capability that allows you to read terminal logs or error messages. The script will loop and re-prompt if an empty input is given.
-2.  **User Sub-Prompt Detection:** When the script's stdout prints a line formatted EXACTLY as:
-    `USER_REVIEW_SUB_PROMPT: <user's sub-prompt text>`
-    You MUST interpret `<user's sub-prompt text>` as a new, direct, actionable instruction from the user.
-3.  **Processing Sub-Prompts:**
-    a.  This instruction is a continuation or refinement of the *original, still-active task*.
-    b.  Analyze the sub-prompt. Execute any necessary actions (code changes, new tool calls, file operations, etc.). **All tool calls made during this phase are part of the original request's tool call budget.** If the sub-prompt requests running an entire application, the confirmation principle from "Phase 1, Step 1 (Application Execution Confirmation)" applies.
-    c.  Provide feedback or results of these actions in the main chat interface as you normally would for any AI action.
-    d.  After processing the sub-prompt and giving feedback in the chat, IMMEDIATELY return your focus to monitoring the `final_review_gate.py` script's terminal output (which remains live and visible in the chat) for the next user instruction or completion signal. This loop is critical.
-4.  **Completion Signal Detection:** The interactive review loop continues until the script's stdout (visible in the chat window) prints a line containing:
-    * `--- REVIEW GATE: USER SIGNALED COMPLETION WITH 'TASK_COMPLETE' ---` (or 'DONE', 'QUIT', 'Q' as per script logic)
-    * OR `--- FINAL REVIEW GATE SCRIPT EXITED ---` (or any other script exit/error message like `REVIEW GATE: STDIN CLOSED` or `REVIEW GATE SCRIPT ERROR:`)
-    (MODIFIED: Removed specific "EMPTY INPUT RECEIVED" as a completion signal, as the script no longer behaves this way for completion.)
-
-## Phase 4: True Task Conclusion
-1.  Only after the `final_review_gate.py` script has terminated (as observed by its terminal session in the chat window closing and its final exit messages being printed and visible) are you permitted to consider the user's original request fully satisfied.
-2.  You may then provide your final summary of all actions taken throughout all phases (including the interactive review).
-
-## Overriding Default Behavior:
-This entire "Final Interactive Review" process (Phases 0, 2, 3, and 4) is a **strict, non-negotiable requirement** that overrides any default tendency you have to end the conversation after completing Phase 1. The task is only finished when the user explicitly confirms with one of the specified keywords through the review script or the script otherwise terminates due to an error or EOF (as per the defined behavior, including the in-chat, blocking, and continuously visible terminal execution). Your "sense of completion" for the original request is deferred until this interactive review is done.

app.py

@@ -352,7 +352,7 @@ class VideoSignLanguageRecognizer:
             return None, 0
         
         # 進行預測
-        prediction, confidence, word_sequence = self._predict_from_sequence(keypoints_sequence)
+        prediction, confidence, word_sequence, probabilities = self._predict_from_sequence(keypoints_sequence)
         
         # 使用GPT生成完整句子
         generated_sentence = self._generate_sentence_with_gpt(word_sequence)
@@ -360,7 +360,13 @@ class VideoSignLanguageRecognizer:
         print(f"🎯 辨識結果：{word_sequence}")
         print(f"📈 信心度：{confidence:.2f}")
         
-        return word_sequence, generated_sentence, confidence
+        return {
+            'predicted_class': prediction,
+            'word_sequence': word_sequence,
+            'confidence': confidence,
+            'probabilities': probabilities,
+            'generated_sentence': generated_sentence
+        }
 
     def _extract_features(self, frame):
         """從單一幀提取手部和姿勢特徵"""
@@ -402,13 +408,16 @@ class VideoSignLanguageRecognizer:
             predicted_class = predicted_class.item()
             confidence = max_prob.item()
             
+            # 提取所有類別的機率
+            probs = probabilities[0].cpu().numpy()
+            
         if confidence >= self.threshold:
             predicted_word = self.label_map.get(predicted_class, f"類別{predicted_class}")
             word_sequence = [predicted_word]
         else:
             word_sequence = []
             
-        return predicted_class, confidence, word_sequence
+        return predicted_class, confidence, word_sequence, probs
 
     def _generate_sentence_with_gpt(self, word_sequence):
         """使用GPT根據單詞序列生成一個完整句子"""
@@ -884,7 +893,7 @@ def process_video():
         video_recognizer = VideoSignLanguageRecognizer(model_path, threshold=0.5)
         
         # 處理影片
-        word_sequence, recognition_result, confidence = video_recognizer.process_video(video_path)
+        result = video_recognizer.process_video(video_path)
         
         # 清理臨時檔案
         try:
@@ -892,36 +901,37 @@ def process_video():
         except:
             pass
         
-        if recognition_result is not None:
-            # 如果是來自 Messenger 的請求，直接回傳結果給用戶
-            if sender_id != 'unknown':
-                send_message(sender_id, recognition_result)
+        if result is not None:
+            # 提取結果數據
+            predicted_class = result.get('predicted_class', -1)
+            word_sequence = result.get('word_sequence', [])
+            confidence = result.get('confidence', 0.0)
+            probabilities = result.get('probabilities', [])
+            generated_sentence = result.get('generated_sentence', '無法辨識手語內容')
             
-            # 解析辨識結果，提供完整的前端所需資訊
-            word_sequence = []
+            # 創建類別機率數據供前端使用
+            prob_data = []
+            if len(probabilities) > 0:
+                sorted_indices = np.argsort(probabilities)[::-1][:4]  # 取前4個最高機率
+                for idx in sorted_indices:
+                    prob = float(probabilities[idx])
+                    class_label = video_recognizer.label_map.get(idx, f"類別{idx}")
+                    prob_data.append({"label": class_label, "probability": prob})
             
-            # 嘗試從辨識結果中提取單詞序列（簡單的文字分割）
-            if recognition_result and recognition_result != "無法辨識手語內容":
-                # 如果結果包含多個詞，可以分割
-                if isinstance(recognition_result, list):
-                    potential_words = recognition_result  # 如果是列表，直接使用
-                else:
-                    potential_words = recognition_result.split()  # 如果是字符串，使用 split()
-                    if len(potential_words) <= 4:  # 假設是單詞序列
-                        word_sequence = potential_words
-                    else:
-                        # 否則視為生成的句子
-                        word_sequence = [recognition_result.split()[0]] if recognition_result.split() else []
+            # 獲取預測類別的標籤
+            predicted_label = video_recognizer.label_map.get(predicted_class, "未知") if predicted_class >= 0 else "未知"
             
-            # 使用 GPT 生成完整句子（而不是直接複製辨識結果）
-            video_recognizer = VideoSignLanguageRecognizer(model_path, threshold=0.5)
-            generated_sentence = video_recognizer._generate_sentence_with_gpt(word_sequence)
+            # 如果是來自 Messenger 的請求，發送GPT生成的句子
+            if sender_id != 'unknown':
+                send_message(sender_id, generated_sentence)
             
             return jsonify({
                 "status": "success",
-                "recognition_result": recognition_result,
-                "confidence": float(confidence),
+                "predicted_class": predicted_class,
+                "predicted_label": predicted_label,
                 "word_sequence": word_sequence,
+                "confidence": float(confidence),
+                "probabilities": prob_data,
                 "generated_sentence": generated_sentence,
                 "sender_id": sender_id
             })
@@ -1035,7 +1045,7 @@ def process_messenger_video(video_url, sender_id):
         video_recognizer = VideoSignLanguageRecognizer(model_path, threshold=0.5)
         
         # 處理影片
-        word_sequence, recognition_result, confidence = video_recognizer.process_video(file_path)
+        result = video_recognizer.process_video(file_path)
         
         # 清理臨時檔案
         try:
@@ -1043,13 +1053,18 @@ def process_messenger_video(video_url, sender_id):
         except:
             pass
         
-        if recognition_result:
+        if result:
+            generated_sentence = result.get('generated_sentence', '無法辨識手語內容')
+            confidence = result.get('confidence', 0.0)
+            word_sequence = result.get('word_sequence', [])
+            
             print(f"✅ 手語辨識完成 - 用戶：{sender_id}")
-            print(f"📝 辨識結果：{recognition_result}")
+            print(f"📝 模型辨識：{word_sequence}")
+            print(f"💬 GPT翻譯：{generated_sentence}")
             print(f"🎯 信心度：{confidence:.2f}")
             
-            # 發送結果給用戶
-            send_message(sender_id, recognition_result)
+            # 發送GPT翻譯結果給用戶
+            send_message(sender_id, generated_sentence)
         else:
             send_message(sender_id, "抱歉，無法辨識您的手語內容，請再試一次。")
             

templates/index.html

@@ -1472,9 +1472,9 @@
                     // 完成進度
                     updateProgress(100, '神經網路分析完成！');
                     
-                    // 顯示辨識結果標籤
-                    const recognitionText = result.recognition_result || '神經網路分析完成';
-                    if (resultLabel) resultLabel.textContent = recognitionText;
+                    // 顯示當前預測 (模型預測的類別)
+                    const predictedLabel = result.predicted_label || '未知';
+                    if (resultLabel) resultLabel.textContent = predictedLabel;
                     
                     // 顯示信心度
                     const confidence = result.confidence || 0;
@@ -1491,25 +1491,46 @@
                         });
                     }
                     
-                    // 顯示單詞序列
+                    // 更新類別機率顯示
+                    if (result.probabilities && probabilitiesContainer) {
+                        probabilitiesContainer.innerHTML = '';
+                        result.probabilities.forEach(function(item) {
+                            const probContainer = document.createElement('div');
+                            probContainer.className = 'mb-3';
+                            
+                            const probLabel = document.createElement('div');
+                            probLabel.className = 'prob-label';
+                            probLabel.innerHTML = `<span>${item.label}</span><span>${(item.probability * 100).toFixed(1)}%</span>`;
+                            
+                            const barContainer = document.createElement('div');
+                            barContainer.className = 'prob-bar-container';
+                            
+                            const bar = document.createElement('div');
+                            bar.className = 'prob-bar';
+                            bar.style.width = `${item.probability * 100}%`;
+                            
+                            barContainer.appendChild(bar);
+                            probContainer.appendChild(probLabel);
+                            probContainer.appendChild(barContainer);
+                            probabilitiesContainer.appendChild(probContainer);
+                        });
+                    }
+                    
+                    // 顯示辨識結果 (模型識別的單詞序列)
                     if (videoWordSequenceDisplay) {
                         if (result.word_sequence && result.word_sequence.length > 0) {
                             videoWordSequenceDisplay.textContent = result.word_sequence.join(' ');
-                        } else if (result.recognition_result) {
-                            videoWordSequenceDisplay.textContent = result.recognition_result;
                         } else {
-                            videoWordSequenceDisplay.textContent = '無辨識結果';
+                            videoWordSequenceDisplay.textContent = '低於辨識閾值';
                         }
                     }
                     
-                    // 顯示翻譯句子
+                    // 顯示AI翻譯結果 (GPT生成的句子)
                     if (videoSentenceDisplay) {
                         if (result.generated_sentence) {
                             videoSentenceDisplay.textContent = result.generated_sentence;
-                        } else if (result.recognition_result) {
-                            videoSentenceDisplay.textContent = result.recognition_result;
                         } else {
-                            videoSentenceDisplay.textContent = '無翻譯結果';
+                            videoSentenceDisplay.textContent = '無法生成翻譯';
                         }
                     }
                     
@@ -1524,6 +1545,9 @@
                     if (resultConfidence) resultConfidence.textContent = '信心度: 0%';
                     if (videoWordSequenceDisplay) videoWordSequenceDisplay.textContent = '分析失敗';
                     if (videoSentenceDisplay) videoSentenceDisplay.textContent = '分析失敗';
+                    if (probabilitiesContainer) {
+                        probabilitiesContainer.innerHTML = '<div class="metric-label" style="text-align: center; color: var(--text-tertiary);">分析失敗</div>';
+                    }
                 }
             }