Improve URL normalization logic and add API docs

API_DOCUMENTATION.md

@@ -0,0 +1,136 @@
+# Antigravity SEO Scaler - API Documentation
+
+Base URL: `http://localhost:7860` (or your deployed URL)
+
+## 1. Start a New Scan
+Initiates an asynchronous crawling and analysis job for a target domain.
+
+- **Endpoint**: `/api/scanstart`
+- **Method**: `POST`
+- **Headers**: `Content-Type: application/json`
+- **Body Parameters**:
+    - `domain` (string, required): The URL of the website to scan (e.g., "https://example.com").
+    - `limit` (integer, optional): Maximum number of pages to crawl. Default is `25`.
+
+**Example Request (curl):**
+```bash
+curl -X POST "http://localhost:7860/api/scanstart" \
+     -H "Content-Type: application/json" \
+     -d '{"domain": "https://example.com", "limit": 50}'
+```
+
+**Example Response:**
+```json
+{
+  "job_id": "8e8c40c5-77e0-4ad6-906a-53d038cd9fe7"
+}
+```
+
+---
+
+## 2. Check Scan Progress
+Retrieves the real-time status of a running job. You can use either a path parameter or a query parameter.
+
+### Option A: Path Parameter
+- **Endpoint**: `/api/progress/{job_id}`
+- **Method**: `GET`
+
+### Option B: Query Parameter
+- **Endpoint**: `/api/progress`
+- **Method**: `GET`
+- **Query Parameters**:
+    - `job_id` (string, required): The ID returned by the start endpoint.
+
+**Example Request:**
+```bash
+curl "http://localhost:7860/api/progress/8e8c40c5-77e0-4ad6-906a-53d038cd9fe7"
+# OR
+curl "http://localhost:7860/api/progress?job_id=8e8c40c5-77e0-4ad6-906a-53d038cd9fe7"
+```
+
+**Example Response:**
+```json
+{
+  "status": "running",
+  "percent": 45,
+  "pages_scanned": 12,
+  "images_found": 86,
+  "message": "Scanning: https://example.com/about",
+  "elapsed_seconds": 15,
+  "eta_seconds": 20,
+  "error": null
+}
+```
+*Possible Statuses*: `pending`, `running`, `done`, `error`
+
+---
+
+## 3. Get Scan Results
+Retrieves the final detailed JSON report. This should be called when the progress status is `done`.
+
+### Option A: Path Parameter
+- **Endpoint**: `/api/result/{job_id}`
+- **Method**: `GET`
+
+### Option B: Query Parameter
+- **Endpoint**: `/api/result`
+- **Method**: `GET`
+- **Query Parameters**:
+    - `job_id` (string, required): The ID of the completed job.
+
+**Example Request:**
+```bash
+curl "http://localhost:7860/api/result/8e8c40c5-77e0-4ad6-906a-53d038cd9fe7"
+```
+
+**Example Response:**
+```json
+{
+  "summary": {
+    "total_pages_scanned": 50,
+    "total_images_found": 320,
+    "total_images_missing_alt": 45,
+    "blocked_reason": null,
+    "crawl_blocked": false
+  },
+  "details": [
+    {
+      "page_url": "https://example.com",
+      "images": [ ... ]
+    }
+  ]
+}
+```
+
+---
+
+## 4. System Status
+Checks the health and resource usage of the underlying server.
+
+- **Endpoint**: `/api/status`
+- **Method**: `GET`
+- **Query Parameters**:
+    - `domain` (string, optional): A target URL to check reachability for.
+
+**Example Response:**
+```json
+{
+  "cpu_percent": 12.5,
+  "memory_percent": 45.2,
+  "disk_usage": 60.1,
+  "target_reachable": true
+}
+```
+
+---
+
+## 5. Health Check
+Simple endpoint for load balancers or uptime monitors.
+
+- **Endpoint**: `/health`
+- **Method**: `GET`
+
+**Example Response:**
+```json
+{ "status": "alive" }
+```

src/crawler.py

@@ -406,10 +406,17 @@ class Crawler:
                     for href in raw_links:
                         full_url = urljoin(url, href)
                         parsed_url = urlparse(full_url)
-                        full_url = parsed_url._replace(fragment="").geturl()
+                        # Normalize: Remove fragment, strip trailing slash to avoid duplicates (/about vs /about/)
+                        full_url = parsed_url._replace(fragment="").geturl().rstrip('/')
+                        
                         link_domain = parsed_url.netloc
                         
-                        is_internal = link_domain == start_domain or link_domain.endswith('.' + base_domain) or link_domain == base_domain
+                        # Internal Check: Match base domain (handles www/non-www and subdomains)
+                        is_internal = (
+                            link_domain == start_domain or 
+                            link_domain.endswith('.' + base_domain) or 
+                            link_domain == base_domain
+                        )
                         
                         if is_internal:
                             path = parsed_url.path.lower()
@@ -419,7 +426,8 @@ class Crawler:
                                  links_stats["skipped"] += 1
                                  continue
 
-                            exclude_keywords = ['/account', '/login', '/signin', '/signup', '/cart', '/checkout', '/wishlist', '/auth', 'javascript:', 'mailto:']
+                            # Filter out non-content paths
+                            exclude_keywords = ['/account', '/login', '/signin', '/signup', '/cart', '/checkout', '/wishlist', '/auth', 'javascript:', 'mailto:', 'tel:']
                             if any(k in full_url.lower() for k in exclude_keywords):
                                 links_stats["skipped"] += 1
                                 continue