scrapeRL / docs /test /input_dashboard_streaming_test_report.md
NeerajCodz's picture
fix: resolve scraper functionality and plugin issues
54ec9cb
|
raw
history blame
2.99 kB

Input/Dashboard + Live Stream + Endpoint Test Report

Scope

  • Input-first 2-window UX (Input -> Dashboard) with required fields: assets, instructions, output instructions
  • Real-time scrape flow (SSE + websocket broadcast)
  • Session-based scrape lifecycle (/api/scrape/*)
  • Frontend/backend integration through single docker compose up
  • Full endpoint smoke through frontend proxy (http://localhost:3000/api/*)

Environment

  • Runtime: docker compose up --build -d
  • Frontend: http://localhost:3000
  • Backend: http://localhost:8000
  • Health check: GET http://localhost:3000/api/health -> 200

Regression Fixes Applied

Endpoint Previous issue Fix Result
POST /api/agents/plan 500 (PlannerAgent.create_plan missing) Replaced with deterministic valid plan generation in route 200
GET /api/tools/categories 500 response validation mismatch Updated return typing to match actual payload 200
GET /api/providers and GET /api/providers/google 500 (list_models missing on provider impls) Switched provider model retrieval to get_models() 200
GET /api/plugins/categories 404 due dynamic route capture Moved static /categories route before /{plugin_id} 200

10 Manual Scrape Stream Scenarios (Low/Medium/High)

Test Complexity Output Memory Plugins Status
low-json low json on none completed
medium-csv-plugins medium csv on mcp-html, skill-extractor completed
high-markdown high markdown on mcp-browser, proc-json completed
low-text-no-memory low text off none completed
medium-json-multi-assets medium json on mcp-search completed
high-csv-unavailable-plugin high csv on mcp-pdf partial (expected unavailable-plugin warning)
low-json-simple-query low json on none completed
medium-markdown-plugins medium markdown on skill-planner, proc-csv completed
high-text high text on mcp-browser completed
low-csv low csv on none completed

Full Endpoint Smoke Test (Frontend Proxy)

  • Target: http://localhost:3000/api/*
  • Total calls: 60
  • Server errors (5xx): 0
  • Unexpected statuses: 0
  • Covered route groups: health, agents, tasks, episode, memory, providers, plugins, tools, settings, scrape

Integration Checks

  • GET http://localhost:3000/favicon.ico -> 200 (favicon 404 resolved)
  • Frontend proxy to backend verified for all dashboard-critical endpoints:
    • /api/health
    • /api/agents/list
    • /api/plugins
    • /api/memory/stats/overview
    • /api/settings

Outcome

  • Frontend and backend are now reliably connected via docker compose.
  • The previously failing 500/404 dashboard endpoints are fixed.
  • Input-first session-based scraper flow, live updates, plugins, memory, and scrape lifecycle endpoints are working end-to-end.