Add file path removal, fix progress tracking, add Assets sidebar

FIXES_SUMMARY.md

@@ -1,232 +0,0 @@
-# Fixes Summary - Model Metrics & UX Improvements
-
-## Issues Fixed
-
-### 1. ✅ Best Model Metrics Showing 0.0000 (HIGH PRIORITY)
-
-**Problem:** 
-- Enhanced summary displayed `R² Score: 0.0000, RMSE: 0.0000, MAE: 0.0000` 
-- Backend logs showed correct values: R²=0.713, RMSE=0.207
-
-**Root Cause:**
-The `_generate_enhanced_summary()` method in `src/orchestrator.py` was extracting metrics incorrectly:
-```python
-best_model_data = models_data.get(best_model_name, {})
-metrics["best_model"] = {
-    "r2_score": best_model_data.get("r2", 0),  # ❌ Wrong! Metrics not at top level
-}
-```
-
-The actual structure from `train_baseline_models` is:
-```python
-{
-    "models": {
-        "xgboost": {
-            "test_metrics": {
-                "r2": 0.713,
-                "rmse": 0.207,
-                "mae": 0.15
-            }
-        }
-    }
-}
-```
-
-**Fix:**
-Updated lines 960-988 in `src/orchestrator.py`:
-```python
-best_model_data = models_data.get(best_model_name, {})
-test_metrics = best_model_data.get("test_metrics", {})  # ✅ Access nested test_metrics
-
-metrics["best_model"] = {
-    "name": best_model_name,
-    "r2_score": test_metrics.get("r2", 0),      # ✅ Now gets correct value
-    "rmse": test_metrics.get("rmse", 0),
-    "mae": test_metrics.get("mae", 0)
-}
-```
-
----
-
-### 2. ✅ Missing Baseline Model Comparison (HIGH PRIORITY)
-
-**Problem:**
-- Only showing final tuned XGBoost model
-- Not displaying comparison of all baseline models (Logistic Regression, Random Forest, XGBoost, etc.) before tuning
-- User couldn't see which baseline model performed best
-
-**Fix:**
-Enhanced summary formatting in `src/orchestrator.py` (lines 1088-1132):
-
-**Before:**
-```
-### 🏆 Best Model Performance
-- Model: xgboost
-- R² Score: 0.7130
-```
-
-**After:**
-```
-### 🔬 Baseline Models Comparison
-
-🏆 **Xgboost**: R²=0.7130, RMSE=0.2070, MAE=0.1500
-   **Random Forest**: R²=0.6850, RMSE=0.2180, MAE=0.1620
-   **Lightgbm**: R²=0.6720, RMSE=0.2250, MAE=0.1680
-   **Ridge**: R²=0.5420, RMSE=0.2890, MAE=0.2150
-   **Lasso**: R²=0.5230, RMSE=0.2950, MAE=0.2200
-   **Catboost**: R²=0.4950, RMSE=0.3100, MAE=0.2320
-
-### ⚙️ Hyperparameter Tuning Results
-- Model Type: xgboost
-- Optimized Score: 0.7150
-```
-
-Now shows:
-- ✅ All baseline models sorted by R² score (descending)
-- ✅ Best model highlighted with 🏆 emoji
-- ✅ Clear comparison before showing tuned results
-- ✅ Separate sections for baseline vs tuned models
-
----
-
-### 3. ✅ Poor Formatting with Ugly Code Blocks (MEDIUM PRIORITY)
-
-**Problem:**
-- LLM responses included file paths like `./outputs/data/cleaned.csv`
-- Markdown code blocks appearing in structured data
-- Messy formatting that wasn't aesthetic
-
-**Fix:**
-Strengthened system prompt in `src/orchestrator.py` (lines 408-418):
-
-```python
-**CRITICAL: User Interface Integration & Response Formatting**
-- The user interface automatically displays clickable buttons for all generated plots, reports, and outputs
-- **NEVER mention file paths** (e.g., "./outputs/plots/...", "./outputs/data/...", etc.) in your responses
-- **NEVER use markdown code blocks** for file paths or structured data in final summaries
-- DO NOT say "Output File: ..." or "Saved to: ..." - users can click buttons to view outputs
-- Simply describe what was created and what insights it shows
-- Use clean, aesthetic formatting with proper sections, bullet points, and spacing
-```
-
-**Changes:**
-- ❌ Removed: "Output File: `./outputs/plots/heatmap.html`"
-- ✅ Replaced with: "Generated an interactive correlation heatmap showing relationships between variables"
-- ❌ Removed: "Saved cleaned data to: `./outputs/data/cleaned.csv`"
-- ✅ Replaced with: "Cleaned the dataset by handling missing values and outliers"
-
----
-
-### 4. ✅ No Progress Indicators (MEDIUM PRIORITY)
-
-**Problem:**
-- Long-running workflows had no visibility for users
-- Users couldn't see which step the agent was on
-- No way to know if the system was stuck or processing
-
-**Fix:**
-
-**Backend (`src/orchestrator.py`):**
-1. Added `progress_callback` parameter to `__init__` (lines 137-159)
-2. Updated `_execute_tool()` to report progress (lines 1194-1200):
-   ```python
-   # Report progress before executing
-   if self.progress_callback:
-       self.progress_callback(tool_name, "running")
-   
-   # ... execute tool ...
-   
-   # Report completion
-   if self.progress_callback:
-       self.progress_callback(tool_name, "completed")
-   ```
-
-**API (`src/api/app.py`):**
-1. Added global `progress_store` dict (line 45)
-2. Created `/api/progress/{session_id}` endpoint (lines 88-93)
-3. Updated `/run` endpoint to track progress (lines 244-258):
-   ```python
-   def progress_callback(tool_name: str, status: str):
-       progress_store[session_key].append({
-           "tool": tool_name,
-           "status": status,
-           "timestamp": time.time()
-       })
-   ```
-4. Return progress in response (line 296)
-
-**Frontend (`FRRONTEEEND/components/ChatInterface.tsx`):**
-1. Added `currentStep` state (line 48)
-2. Display progress in typing indicator (lines 531-555):
-   ```tsx
-   {currentStep ? (
-     <div className="flex items-center gap-3">
-       <div className="flex gap-1">
-         <span className="w-1.5 h-1.5 bg-emerald-500 rounded-full animate-bounce"></span>
-       </div>
-       <span className="text-sm text-white/60">
-         🔧 {currentStep.replace(/_/g, ' ').replace('train', 'Training')...}
-       </span>
-     </div>
-   ) : (
-     // Default loading animation
-   )}
-   ```
-
-**Result:**
-- ✅ User sees: "🔧 Training Baseline Models..." while models train
-- ✅ User sees: "🔧 Cleaning Missing Values..." during data cleaning
-- ✅ User sees: "🔧 Generating Plotly Dashboard..." during visualization
-- ✅ Clear visibility of current step throughout workflow
-- ✅ Emerald-colored animated dots indicate active processing
-
----
-
-## Testing Recommendations
-
-1. **Metric Extraction:**
-   - Upload earthquake dataset
-   - Run full ML pipeline
-   - Verify metrics display correctly (not 0.0000)
-
-2. **Baseline Comparison:**
-   - Check that all models appear in summary
-   - Verify sorting by R² score
-   - Confirm best model has 🏆 emoji
-
-3. **Formatting:**
-   - Check that no file paths appear in responses
-   - Verify clean markdown without code blocks for structured data
-
-4. **Progress Indicators:**
-   - Upload large dataset
-   - Watch for step-by-step progress updates
-   - Confirm smooth transition when complete
-
-## Files Modified
-
-1. `src/orchestrator.py` (4 changes)
-   - Lines 137-159: Added `progress_callback` parameter
-   - Lines 960-988: Fixed metric extraction from `test_metrics`
-   - Lines 1088-1132: Added baseline model comparison section
-   - Lines 408-418: Strengthened formatting rules
-   - Lines 1194-1200, 1248-1258: Added progress reporting
-
-2. `src/api/app.py` (4 changes)
-   - Line 7: Import `time`
-   - Line 45: Added `progress_store` dict
-   - Lines 88-93: Created `/api/progress/{session_id}` endpoint
-   - Lines 170-185, 244-258, 296: Integrated progress callback
-
-3. `FRRONTEEEND/components/ChatInterface.tsx` (3 changes)
-   - Line 48: Added `currentStep` state
-   - Line 140: Clear progress on response
-   - Lines 531-555: Enhanced typing indicator with progress display
-
-## Impact
-
-- ✅ Model metrics now display correctly (not 0.0000)
-- ✅ Users can see all baseline models before tuning results
-- ✅ Responses are cleaner without file paths/ugly code blocks
-- ✅ Real-time progress visibility improves UX significantly
-- ✅ Users won't think the system is stuck during long operations

FRRONTEEEND/components/ChatInterface.tsx

@@ -1,7 +1,7 @@
 
 import React, { useState, useRef, useEffect } from 'react';
 import { motion, AnimatePresence } from 'framer-motion';
-import { Send, Plus, Search, Settings, MoreHorizontal, User, Bot, ArrowLeft, Paperclip, Sparkles, Trash2, X, Upload } from 'lucide-react';
+import { Send, Plus, Search, Settings, MoreHorizontal, User, Bot, ArrowLeft, Paperclip, Sparkles, Trash2, X, Upload, Package, FileText, BarChart3, ChevronRight } from 'lucide-react';
 import { cn } from '../lib/utils';
 import { Logo } from './Logo';
 import ReactMarkdown from 'react-markdown';
@@ -48,6 +48,7 @@ export const ChatInterface: React.FC<{ onBack: () => void }> = ({ onBack }) => {
   const [currentStep, setCurrentStep] = useState<string>('');
   const [uploadedFile, setUploadedFile] = useState<File | null>(null);
   const [reportModalUrl, setReportModalUrl] = useState<string | null>(null);
+  const [showAssets, setShowAssets] = useState(false);
   const fileInputRef = useRef<HTMLInputElement>(null);
   const scrollRef = useRef<HTMLDivElement>(null);
   
@@ -120,6 +121,7 @@ export const ChatInterface: React.FC<{ onBack: () => void }> = ({ onBack }) => {
         if (uploadedFile) {
           formData.append('file', uploadedFile);
           formData.append('task_description', input || 'Analyze this dataset and provide insights');
+          formData.append('session_id', sessionKey); // Add session_id for progress tracking
         } else if (hasRecentFile) {
           // For follow-up questions, extract the filename from recent context
           const fileNameMatch = recentFileMessage?.content.match(/Uploaded: (.+)/);
@@ -127,7 +129,7 @@ export const ChatInterface: React.FC<{ onBack: () => void }> = ({ onBack }) => {
           
           // Send follow-up request as a new task description
           formData.append('task_description', input);
-          formData.append('session_id', activeSessionId);
+          formData.append('session_id', sessionKey); // Use same session key
           
           // Note: Backend needs to support session-based file context
           // For now, just send the task which should work with session memory
@@ -426,6 +428,15 @@ export const ChatInterface: React.FC<{ onBack: () => void }> = ({ onBack }) => {
              </div>
           </div>
           <div className="flex items-center gap-3">
+            <button 
+              onClick={() => setShowAssets(!showAssets)}
+              className={cn(
+                "p-2 transition-colors rounded-lg",
+                showAssets ? "text-emerald-400 bg-emerald-500/10" : "text-white/40 hover:text-white"
+              )}
+            >
+              <Package className="w-5 h-5" />
+            </button>
             <button className="p-2 text-white/40 hover:text-white transition-colors">
               <Search className="w-5 h-5" />
             </button>
@@ -659,6 +670,138 @@ export const ChatInterface: React.FC<{ onBack: () => void }> = ({ onBack }) => {
         </div>
       </main>
       
+      {/* Assets Sidebar */}
+      <AnimatePresence>
+        {showAssets && (
+          <motion.aside
+            initial={{ x: 320, opacity: 0 }}
+            animate={{ x: 0, opacity: 1 }}
+            exit={{ x: 320, opacity: 0 }}
+            transition={{ type: "spring", damping: 25, stiffness: 200 }}
+            className="w-[320px] border-l border-white/5 bg-[#0a0a0a]/95 backdrop-blur-xl flex flex-col"
+          >
+            <div className="p-4 border-b border-white/5 flex items-center justify-between">
+              <div className="flex items-center gap-2">
+                <Package className="w-5 h-5 text-emerald-400" />
+                <h3 className="font-bold text-sm">Assets</h3>
+              </div>
+              <button 
+                onClick={() => setShowAssets(false)}
+                className="p-1.5 rounded-lg hover:bg-white/5 transition-colors"
+              >
+                <X className="w-4 h-4" />
+              </button>
+            </div>
+            
+            <div className="flex-1 overflow-y-auto p-4 space-y-4 custom-scrollbar">
+              {/* Collect all assets from messages */}
+              {(() => {
+                const allPlots: Array<{title: string, url: string, type?: string}> = [];
+                const allReports: Array<{name: string, path: string}> = [];
+                const allModels: string[] = [];
+                
+                activeSession.messages.forEach(msg => {
+                  if (msg.plots) allPlots.push(...msg.plots);
+                  if (msg.reports) allReports.push(...msg.reports);
+                  // Extract model references from content
+                  if (msg.content.includes('xgboost') || msg.content.includes('model')) {
+                    const modelMatch = msg.content.match(/\b(xgboost|random_forest|catboost|lightgbm)[^\s]*/gi);
+                    if (modelMatch) allModels.push(...modelMatch);
+                  }
+                });
+                
+                const uniqueModels = [...new Set(allModels)];
+                
+                return (
+                  <>
+                    {/* Models Section */}
+                    {uniqueModels.length > 0 && (
+                      <div>
+                        <div className="flex items-center gap-2 mb-3">
+                          <FileText className="w-4 h-4 text-blue-400" />
+                          <h4 className="text-xs font-bold uppercase tracking-wider text-white/60">Models ({uniqueModels.length})</h4>
+                        </div>
+                        <div className="space-y-2">
+                          {uniqueModels.slice(0, 5).map((model, idx) => (
+                            <div 
+                              key={idx}
+                              className="p-3 rounded-lg bg-white/5 border border-white/10 hover:bg-white/10 transition-all cursor-pointer group"
+                            >
+                              <div className="flex items-center justify-between">
+                                <span className="text-sm text-white/80 truncate flex-1">{model}</span>
+                                <ChevronRight className="w-4 h-4 text-white/40 group-hover:text-white/80 transition-all" />
+                              </div>
+                            </div>
+                          ))}
+                        </div>
+                      </div>
+                    )}
+                    
+                    {/* Plots Section */}
+                    {allPlots.length > 0 && (
+                      <div>
+                        <div className="flex items-center gap-2 mb-3">
+                          <BarChart3 className="w-4 h-4 text-emerald-400" />
+                          <h4 className="text-xs font-bold uppercase tracking-wider text-white/60">Visualizations ({allPlots.length})</h4>
+                        </div>
+                        <div className="space-y-2">
+                          {allPlots.map((plot, idx) => (
+                            <button
+                              key={idx}
+                              onClick={() => setReportModalUrl(plot.url)}
+                              className="w-full p-3 rounded-lg bg-white/5 border border-white/10 hover:bg-emerald-500/10 hover:border-emerald-500/30 transition-all text-left group"
+                            >
+                              <div className="flex items-center justify-between">
+                                <span className="text-sm text-white/80 truncate flex-1">{plot.title}</span>
+                                <ChevronRight className="w-4 h-4 text-white/40 group-hover:text-emerald-400 transition-all" />
+                              </div>
+                              <span className="text-xs text-white/40 mt-1 block">{plot.type || 'interactive'}</span>
+                            </button>
+                          ))}
+                        </div>
+                      </div>
+                    )}
+                    
+                    {/* Reports Section */}
+                    {allReports.length > 0 && (
+                      <div>
+                        <div className="flex items-center gap-2 mb-3">
+                          <FileText className="w-4 h-4 text-purple-400" />
+                          <h4 className="text-xs font-bold uppercase tracking-wider text-white/60">Reports ({allReports.length})</h4>
+                        </div>
+                        <div className="space-y-2">
+                          {allReports.map((report, idx) => (
+                            <button
+                              key={idx}
+                              onClick={() => setReportModalUrl(report.path)}
+                              className="w-full p-3 rounded-lg bg-white/5 border border-white/10 hover:bg-purple-500/10 hover:border-purple-500/30 transition-all text-left group"
+                            >
+                              <div className="flex items-center justify-between">
+                                <span className="text-sm text-white/80 truncate flex-1">{report.name}</span>
+                                <ChevronRight className="w-4 h-4 text-white/40 group-hover:text-purple-400 transition-all" />
+                              </div>
+                            </button>
+                          ))}
+                        </div>
+                      </div>
+                    )}
+                    
+                    {/* Empty State */}
+                    {allPlots.length === 0 && allReports.length === 0 && uniqueModels.length === 0 && (
+                      <div className="flex flex-col items-center justify-center h-full text-center p-8">
+                        <Package className="w-12 h-12 text-white/10 mb-3" />
+                        <p className="text-sm text-white/40 mb-1">No assets yet</p>
+                        <p className="text-xs text-white/30">Upload a dataset to generate visualizations and models</p>
+                      </div>
+                    )}
+                  </>
+                );
+              })()}
+            </div>
+          </motion.aside>
+        )}
+      </AnimatePresence>
+      
       {/* Report Modal */}
       <AnimatePresence>
         {reportModalUrl && (

src/orchestrator.py

@@ -2304,8 +2304,26 @@ You are a DOER. Complete workflows based on user intent."""
                             task_description
                         )
                         summary_text = enhanced_summary["text"]
+                        
+                        # 🧹 POST-PROCESS: Remove any file paths that slipped through
+                        import re
+                        # Remove file path patterns
+                        summary_text = re.sub(r'\./outputs/[^\s\)]+', '[generated file]', summary_text)
+                        summary_text = re.sub(r'/outputs/[^\s\)]+', '[generated file]', summary_text)
+                        summary_text = re.sub(r'outputs/[^\s\)]+', '[generated file]', summary_text)
+                        # Remove common file path mentions
+                        summary_text = re.sub(r'saved to:?\s*[^\s]+', 'generated', summary_text, flags=re.IGNORECASE)
+                        summary_text = re.sub(r'output file:?\s*[^\s]+', 'output generated', summary_text, flags=re.IGNORECASE)
+                        summary_text = re.sub(r'file path:?\s*[^\s]+', 'file generated', summary_text, flags=re.IGNORECASE)
+                        # Remove filename patterns in parentheses and backticks
+                        summary_text = re.sub(r'\([^\)]*\.(csv|pkl|html|png|json)[^\)]*\)', '', summary_text)
+                        summary_text = re.sub(r'`[^`]*\.(csv|pkl|html|png|json)[^`]*`', '', summary_text)
+                        # Clean up table separators that mention paths
+                        summary_text = re.sub(r'\|\s*[^\|]*\.(csv|pkl|html|png)[^\|]*\s*\|', '| [see artifacts] |', summary_text)
+                        
                         metrics_data = enhanced_summary.get("metrics", {})
                         artifacts_data = enhanced_summary.get("artifacts", {})
+                        artifacts_data = enhanced_summary.get("artifacts", {})
                         plots_data = enhanced_summary.get("plots", [])
                         print(f"✅ Enhanced summary generated with {len(plots_data)} plots, {len(metrics_data)} metrics")
                     except Exception as e: