docs/CONCEPTION_DESIGN_DOCUMENT.md

# IntelliScan — Conception & Design Document

**Project:** IntelliScan — AI-Powered Web Vulnerability Scanner  
**Version:** 1.0.0  
**Author:** SABKARI Mohamed  
**Institution:** Ibn Tofail University, Kenitra, Morocco  
**Academic Context:** Master's Thesis (PFE) — Cybersecurity & AI, 2025–2026  
**Supervisor:** Pr. Youssef FAKHRI  
**License:** MIT  
**Date:** May 2026  

---

## Table of Contents

1. [Project Overview](#1-project-overview)  
2. [Functional Requirements](#2-functional-requirements)  
3. [System Architecture](#3-system-architecture)  
4. [Detailed Module Design](#4-detailed-module-design)  
5. [ML Classifier Design](#5-ml-classifier-design)  
6. [Database / Persistence Design](#6-database--persistence-design)  
7. [Interface Design](#7-interface-design)  
8. [Security & Ethical Design](#8-security--ethical-design)  
9. [Test Strategy](#9-test-strategy)  
10. [Deployment Design](#10-deployment-design)  
11. [Technology Choices Justification](#11-technology-choices-justification)  
12. [Limitations & Future Work](#12-limitations--future-work)  

---

## 1. Project Overview

### 1.1 Problem Statement

Web application vulnerabilities remain one of the most exploited attack vectors in cybersecurity. According to the OWASP Top 10 (2021), injection flaws (SQLi), Cross-Site Scripting (XSS), and insecure file handling (LFI) consistently rank among the most critical threats. Existing vulnerability scanners present several limitations:

| Scanner | Limitation |
|---------|-----------|
| **OWASP ZAP** | Complex UI, steep learning curve, no ML-based detection |
| **Burp Suite** | Commercial license required for advanced features |
| **Nikto** | Signature-only, no adaptive classification |
| **SQLMap** | Single vulnerability type (SQLi only) |
| **w3af** | Discontinued maintenance, outdated engine |

**IntelliScan** addresses these gaps by combining **signature-based detection** with **machine learning classification** in a unified, open-source, modular pipeline. The dual-approach design provides both the reliability of known-pattern matching and the adaptability of behavioral ML classification.

### 1.2 Objectives and Scope

**Primary Objectives:**
1. Automate authenticated crawling and vulnerability detection for SQLi, XSS, and LFI
2. Provide a dual detection engine (signature + Random Forest ML) to improve detection confidence
3. Generate professional PDF reports suitable for stakeholder communication
4. Offer a modular, extensible architecture enabling future vulnerability type additions

**Scope Boundaries:**
- **In scope:** SQLi, reflected XSS, stored XSS, LFI on HTTP/HTTPS targets
- **Out of scope (v1.0):** CSRF, SSRF, command injection, blind SQLi, DOM XSS, API fuzzing

### 1.3 Target Users

| Actor | Profile | Use Case |
|-------|---------|----------|
| **Penetration Tester** | Security professional with CLI experience | Authorized vulnerability assessments |
| **Developer** | Web developer integrating security into CI/CD | Pre-deployment security checks |
| **Academic Researcher** | Student/professor studying ML-based detection | Benchmarking detection algorithms |
| **Security Auditor** | Compliance officer reviewing application security | Generating audit evidence (PDF reports) |

### 1.4 Ethical and Legal Framework

IntelliScan is designed exclusively for **authorized security testing**. The tool enforces:

- **Explicit consent requirement:** Users must have written authorization before scanning
- **Identification header:** All HTTP requests include `User-Agent: IntelliScan/1.0`
- **Rate limiting:** 0.1s delay between requests to minimize target impact
- **Scope restriction:** Crawler enforces same-domain policy and blacklists sensitive paths

**Legal compliance:**
- **Morocco — Loi 09-08:** Data protection for scan results containing PII
- **EU — RGPD/GDPR:** Applicable when scanning EU-hosted targets
- **US — CFAA:** Unauthorized access prohibition; IntelliScan requires explicit authorization
- **Responsible Disclosure:** Tool documentation recommends coordinated disclosure practices

---

## 2. Functional Requirements

### 2.1 Use Cases

```
┌──────────────────────────────────────────────────────────────┐
│                    IntelliScan System                        │
│                                                              │
│  ┌─────────┐    ┌──────────────────────────────────────┐     │
│  │ Security │───>│ UC-01: Run Full Scan Pipeline       │     │
│  │ Analyst  │    └──────────────────────────────────────┘     │
│  │          │    ┌──────────────────────────────────────┐     │
│  │          │───>│ UC-02: Crawl Target Only             │     │
│  │          │    └──────────────────────────────────────┘     │
│  │          │    ┌──────────────────────────────────────┐     │
│  │          │───>│ UC-03: Train ML Classifier           │     │
│  │          │    └──────────────────────────────────────┘     │
│  │          │    ┌──────────────────────────────────────┐     │
│  │          │───>│ UC-04: Generate Payload Variants     │     │
│  │          │    └──────────────────────────────────────┘     │
│  │          │    ┌──────────────────────────────────────┐     │
│  │          │───>│ UC-05: View PDF Report               │     │
│  └─────────┘    └──────────────────────────────────────┘     │
│                                                              │
│  ┌─────────┐    ┌──────────────────────────────────────┐     │
│  │ CI/CD   │───>│ UC-06: Automated Scan (CLI)          │     │
│  │ Pipeline│    └──────────────────────────────────────┘     │
│  └─────────┘                                                 │
│                                                              │
│  ┌─────────┐    ┌──────────────────────────────────────┐     │
│  │ Discord │<───│ UC-07: Receive Scan Notification     │     │
│  │ Channel │    └──────────────────────────────────────┘     │
│  └─────────┘                                                 │
└──────────────────────────────────────────────────────────────┘
```

### 2.2 Functional Requirements per Module

#### Module 1 — Crawler

| ID | Requirement | Priority |
|----|-------------|----------|
| FR-CRAWL-01 | Authenticate via POST /login.php with CSRF token extraction | Must |
| FR-CRAWL-02 | Set DVWA security level to "low" automatically | Must |
| FR-CRAWL-03 | BFS traversal with configurable MAX_DEPTH (default=5) | Must |
| FR-CRAWL-04 | Limit to MAX_PAGES (default=200) to prevent infinite crawling | Must |
| FR-CRAWL-05 | Extract HTML forms with action, method, input names/types/values | Must |
| FR-CRAWL-06 | Extract URL query parameters from discovered pages | Must |
| FR-CRAWL-07 | Enforce same-domain policy (no off-site crawling) | Must |
| FR-CRAWL-08 | Blacklist sensitive paths (/logout.php, /setup.php, /.git/, /.env) | Must |
| FR-CRAWL-09 | Output results.json with structured crawl data | Must |

#### Module 2 — Injector

| ID | Requirement | Priority |
|----|-------------|----------|
| FR-INJ-01 | Load payloads from payloads/ directory (sqli.txt, xss.txt, lfi.txt) | Must |
| FR-INJ-02 | Support GET and POST injection methods | Must |
| FR-INJ-03 | Concurrent injection via ThreadPoolExecutor (MAX_CONCURRENT=8) | Must |
| FR-INJ-04 | Auto-fetch CSRF token for POST forms (user_token) | Must |
| FR-INJ-05 | Capture status_code, response_len, response_body (truncated 8000 chars) | Must |
| FR-INJ-06 | Support DVWA_FORMS fallback definitions for 4 endpoints | Should |
| FR-INJ-07 | Output injection_results.json | Must |

#### Module 3 — Analyzer

| ID | Requirement | Priority |
|----|-------------|----------|
| FR-ANAL-01 | Detect SQLi via "first name:" count and SQL_ERROR_PATTERNS (8 patterns) | Must |
| FR-ANAL-02 | Detect XSS via payload reflection check and XSS_DANGEROUS_PATTERNS | Must |
| FR-ANAL-03 | Detect LFI via LFI_SIGNATURES (root:x:0:0, /bin/bash, etc.) | Must |
| FR-ANAL-04 | Assign severity: sqli=HIGH, xss_r=MEDIUM, xss_s=HIGH, lfi=HIGH | Must |
| FR-ANAL-05 | Label each result as VULNERABLE or NOT_VULNERABLE with reason | Must |
| FR-ANAL-06 | Output labeled_results.json | Must |

#### Module 4 — ML Classifier

| ID | Requirement | Priority |
|----|-------------|----------|
| FR-ML-01 | Extract 8 behavioral features (no data leakage) | Must |
| FR-ML-02 | Train RandomForestClassifier (100 trees, balanced, max_depth=10) | Must |
| FR-ML-03 | 80/20 stratified train-test split | Must |
| FR-ML-04 | 5-fold cross-validation with F1-weighted scoring | Must |
| FR-ML-05 | Persist model + mean_response_len_per_type via joblib | Must |
| FR-ML-06 | Require minimum 4 samples to train | Must |
| FR-ML-07 | Produce TrainingReport with metrics and feature importances | Must |

#### Module 5 — Payload Generator

| ID | Requirement | Priority |
|----|-------------|----------|
| FR-GEN-01 | Case alternation mutation (not for LFI) | Must |
| FR-GEN-02 | SQL comment insertion (sqli only) | Must |
| FR-GEN-03 | Quote substitution ' ↔ " (not for LFI) | Must |
| FR-GEN-04 | URL encoding (urllib.parse.quote) | Must |
| FR-GEN-05 | Double URL encoding | Must |
| FR-GEN-06 | Whitespace variation (double space, tab, /**/) | Must |
| FR-GEN-07 | Deduplicate generated variants | Must |

#### Module 6 — Reporter

| ID | Requirement | Priority |
|----|-------------|----------|
| FR-REP-01 | Cover page with target, date, severity summary boxes | Must |
| FR-REP-02 | Executive summary with statistics table by vuln type | Must |
| FR-REP-03 | Detailed findings with severity badge, URL, payload, reason | Must |
| FR-REP-04 | Remediation recommendations (5 per vuln type) | Must |
| FR-REP-05 | Methodology & disclaimer section | Must |
| FR-REP-06 | Output report.pdf using fpdf2 | Must |

### 2.3 Non-Functional Requirements

| ID | Category | Requirement | Target |
|----|----------|-------------|--------|
| NFR-01 | **Performance** | Complete full scan of DVWA in < 5 minutes | < 300s |
| NFR-02 | **Performance** | Concurrent injection with 8 workers | 4-8x speedup |
| NFR-03 | **Reliability** | Retry failed HTTP requests (3 retries, exponential backoff) | 99% delivery |
| NFR-04 | **Security** | Rate limit requests to 0.1s minimum interval | Responsible scanning |
| NFR-05 | **Portability** | Run on Python 3.10+ across Windows, Linux, macOS | Cross-platform |
| NFR-06 | **Portability** | Docker Compose deployment with DVWA test target | One-command setup |
| NFR-07 | **Maintainability** | Modular pipeline architecture (6 independent modules) | Low coupling |
| NFR-08 | **Maintainability** | >80% unit test coverage | pytest-cov |
| NFR-09 | **Usability** | Rich CLI output with colored tables and progress | Developer UX |
| NFR-10 | **Extensibility** | JSON intermediate files enable module replacement | Plugin-ready |

## 3. System Architecture

### 3.1 High-Level Pipeline Diagram

```
 ┌─────────────┐     ┌─────────────┐     ┌─────────────┐
 │   TARGET     │     │  PAYLOADS   │     │   CONFIG     │
 │  Web App     │     │ sqli.txt    │     │  config.py   │
 │  (DVWA)      │     │ xss.txt     │     │  Constants   │
 └──────┬───────┘     │ lfi.txt     │     └──────┬───────┘
        │             └──────┬──────┘            │
        ▼                    │                   │
 ┌──────────────┐            │         ┌─────────▼────────┐
 │  MODULE 1    │            │         │   HttpClient     │
 │  Crawler     │◄───────────┼─────────│   (Session,      │
 │  BFS + Auth  │            │         │    Retry, CSRF)  │
 └──────┬───────┘            │         └──────────────────┘
        │                    │
        ▼ results.json       │
 ┌──────────────┐            │
 │  MODULE 2    │◄───────────┘
 │  Injector    │
 │  GET / POST  │
 └──────┬───────┘
        │
        â–¼ injection_results.json
 ┌──────────────┐
 │  MODULE 3    │
 │  Analyzer    │
 │  Signatures  │
 └──────┬───────┘
        │
        â–¼ labeled_results.json
   ┌────┴─────────────────┐
   │                      │
   â–¼                      â–¼
 ┌──────────────┐  ┌──────────────┐
 │  MODULE 4    │  │  MODULE 6    │
 │  ML Class.   │  │  Reporter    │
 │  Rand Forest │  │  PDF (fpdf2) │
 └──────┬───────┘  └──────┬───────┘
        │                 │
        â–¼                 â–¼
   model.pkl         report.pdf

 ┌──────────────┐
 │  MODULE 5    │ (independent — can run standalone)
 │  Payload Gen │
 │  6 Mutations │
 └──────┬───────┘
        │
        â–¼
  all_payloads.json
```

> **Note:** Generate high-resolution PNG versions with `python docs/generate_diagrams.py`

### 3.2 Module Dependency Graph

```
                    ┌─────────────────┐
                    │   __main__.py   │
                    │   CLI (Click)   │
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │    core.py      │
                    │  IntelliScan    │
                    └────────┬────────┘
                             │
         ┌───────┬───────┬───┼───┬───────┬───────┐
         ▼       ▼       ▼   │   ▼       ▼       ▼
     crawler  injector  anal │  classif  pgen  reporter
       .py     .py      .py │   .py     .py    .py
         │       │       │   │   │       │       │
         ▼       ▼       │   │   │       │       │
      http_client.py     │   │   │       │       │
                         â–¼   â–¼   â–¼       â–¼       â–¼
                      ┌──────────────────────────┐
                      │       config.py          │
                      │  (All constants, Final)  │
                      └──────────────────────────┘
```

### 3.3 Component Interaction Description

The **IntelliScan** orchestrator (`core.py`) implements the **Pipeline** design pattern:

1. **Crawler** authenticates and explores the target via BFS, producing a site map
2. **Injector** reads discovered forms/params and delivers payloads concurrently
3. **Analyzer** applies signature-based heuristics to label each injection result
4. **ML Classifier** trains a Random Forest on the labeled data for secondary classification
5. **Payload Generator** produces WAF-bypass variants (runs independently)
6. **Reporter** compiles all findings into a professional PDF report

**Key design principles:**
- **Loose coupling:** Modules communicate exclusively via JSON files
- **Single responsibility:** Each module has one clearly defined role
- **Fail-safe orchestration:** ML training is skipped if <4 samples; individual module failures don't crash the pipeline

---

## 4. Detailed Module Design

### 4.1 Module 1 — Crawler (`crawler.py`)

**Purpose:** Authenticated BFS exploration of the target web application to discover all injectable forms and URL parameters.

**Input/Output:**

| Direction | Data | Format |
|-----------|------|--------|
| Input | Target URL, optional credentials (user:password) | CLI arguments |
| Output | Site map with forms and URL params | `results.json` |

**Key Algorithms:**
- **BFS traversal** using `collections.deque` with `(url, depth)` tuples
- **CSRF token extraction** via BeautifulSoup parsing of `input[name=user_token]`
- **Form extraction** parsing `<form>`, `<input>`, `<textarea>`, `<select>` elements
- **Same-domain enforcement** via `urlparse().netloc` comparison

**Key Classes:**

| Class | Fields | Role |
|-------|--------|------|
| `Form` | url, action, method, inputs | Represents a discovered HTML form |
| `UrlParam` | url, params | Represents a URL with query parameters |
| `CrawlResult` | target, pages_visited, forms, url_params | Aggregated crawl output |
| `Crawler` | target, auth, max_pages, max_depth, http | Main crawler engine |

**Error Handling:**
- HTTP errors during page fetching are logged and skipped (BFS continues)
- Authentication failure raises `RuntimeError` with clear message
- DVWA security configuration failure is silently ignored (target may not be DVWA)

**Testability:**
- Mock `HttpClient` to provide canned HTML responses
- Test BFS depth limiting, blacklist enforcement, form extraction independently

---

### 4.2 Module 2 — Injector (`injector.py`)

**Purpose:** Inject attack payloads into all discovered forms and URL parameters, capturing HTTP responses for analysis.

**Input/Output:**

| Direction | Data | Format |
|-----------|------|--------|
| Input | Crawl results, payload files | `results.json`, `payloads/*.txt` |
| Output | Injection results with responses | `injection_results.json` |

**Key Algorithms:**
- **Task generation:** Cartesian product of `(vuln_type × form × payload)`
- **Concurrent execution:** `ThreadPoolExecutor` with `MAX_CONCURRENT=8` workers
- **CSRF handling:** Auto-fetches fresh token for each POST injection

**Payload Counts (base):**

| File | Payloads | Examples |
|------|----------|----------|
| `sqli.txt` | 13 | `' OR 1=1 --`, `' UNION SELECT`, `'; DROP TABLE` |
| `xss.txt` | 12 | `<script>alert(1)</script>`, `<img onerror=...>` |
| `lfi.txt` | 11 | `../../../../etc/passwd`, `php://filter/...` |

**DVWA Target Endpoints:**

| Vuln Type | URL | Method | Parameters |
|-----------|-----|--------|------------|
| sqli | `/vulnerabilities/sqli/` | GET | id |
| xss_r | `/vulnerabilities/xss_r/` | GET | name |
| xss_s | `/vulnerabilities/xss_s/` | POST | txtName, mtxMessage |
| lfi | `/vulnerabilities/fi/` | GET | page |

**Error Handling:**
- Individual injection failures return `None` (skipped in results)
- Response body truncated to 8000 characters to manage memory

---

### 4.3 Module 3 — Analyzer (`analyzer.py`)

**Purpose:** Signature-based labeling of each injection result as VULNERABLE or NOT_VULNERABLE.

**Detection Logic per Type:**

**SQLi Detection:**
```
IF body.count("first name:") >= 1 → VULNERABLE (data dump)
ELSE IF any SQL_ERROR_PATTERNS in body → VULNERABLE (SQL error)
ELSE → NOT_VULNERABLE
```

**SQL_ERROR_PATTERNS (8 signatures):**
1. `you have an error in your sql syntax`
2. `mysql_fetch_array()`
3. `mysql_num_rows()`
4. `supplied argument is not a valid mysql`
5. `warning: mysql_`
6. `unclosed quotation mark`
7. `ora-01756`
8. `syntax error in string in query expression`

**XSS Detection:**
```
IF payload.lower() in response_body → VULNERABLE (reflection)
ELSE IF any XSS_DANGEROUS_PATTERNS reflected → VULNERABLE
ELSE → NOT_VULNERABLE
```

**LFI Detection:**
```
IF any LFI_SIGNATURES in body → VULNERABLE
ELSE → NOT_VULNERABLE
```

**Severity Assignment:**

| Vuln Type | Severity | Rationale |
|-----------|----------|-----------|
| sqli | HIGH | Direct database access, data exfiltration |
| xss_r | MEDIUM | Requires victim interaction (reflected) |
| xss_s | HIGH | Persistent, affects all users |
| lfi | HIGH | Server-side file disclosure, potential RCE |

---

### 4.4 Module 4 — ML Classifier (`classifier.py`)

*(Detailed in Section 5)*

---

### 4.5 Module 5 — Payload Generator (`payload_gen.py`)

**Purpose:** Produce WAF-bypass variants of base payloads via 6 mutation techniques to increase test coverage.

**Mutation Techniques:**

| # | Technique | Example | Applicability |
|---|-----------|---------|---------------|
| 1 | Case alternation | `OR` → `oR`, `Or` | sqli, xss (not lfi) |
| 2 | SQL comment insertion | `OR 1=1` → `OR/**/1=1` | sqli only |
| 3 | Quote substitution | `'` ↔ `"` | sqli, xss (not lfi) |
| 4 | URL encoding | `'` → `%27` | all |
| 5 | Double URL encoding | `'` → `%2527` | all |
| 6 | Whitespace variation | `space` → `tab`, `/**/` | all |

**Coverage Expansion:**

| Type | Base | Generated | Expansion |
|------|------|-----------|-----------|
| sqli | 13 | ~208 | +530% |
| xss | 12 | ~150 | +530% |
| lfi | 11 | ~80 | +300% |

---

### 4.6 Module 6 — Reporter (`reporter.py`)

**Purpose:** Generate a professional 5-section PDF security report using fpdf2.

**Report Structure:**

| Section | Content | Page(s) |
|---------|---------|---------|
| Cover | Target, date, tool name, severity summary boxes | 1 |
| Executive Summary | Statistics table by vuln type, detection rates | 1 |
| Detailed Findings | Per-vulnerability cards with severity badge | 1-N |
| Recommendations | 5 remediation steps per vuln type | 1-2 |
| Methodology | Module descriptions, disclaimer | 1 |

**Severity Color Coding:**

| Severity | RGB Color | Visual |
|----------|-----------|--------|
| HIGH | (192, 57, 43) | Red |
| MEDIUM | (243, 156, 18) | Orange |
| LOW | (241, 196, 15) | Yellow |
| INFO | (39, 174, 96) | Green |

---

## 5. ML Classifier Design

### 5.1 Feature Engineering Rationale

The classifier uses **8 purely behavioral features** extracted from HTTP responses. This design was chosen after a critical revision that identified **data leakage** in the previous feature set.

**Previous features (REMOVED — data leakage):**
- `has_alert` — directly mirrors XSS analyzer decision
- `has_first_name` — directly mirrors SQLi analyzer decision
- `payload_in_response` — directly mirrors XSS reflection check

**Current 8 behavioral features:**

| # | Feature | Description | Rationale |
|---|---------|-------------|-----------|
| F1 | `response_len` | Body length in bytes | Vulnerable responses often contain additional data (SQL dumps, file contents) |
| F2 | `status_code` | HTTP status code | Error codes may indicate injection success/failure |
| F3 | `payload_len` | Length of injected payload | Complex payloads may have different success rates |
| F4 | `vuln_type_encoded` | sqli=0, xss=1, lfi=2 | Different vuln types have different response patterns |
| F5 | `response_to_payload_ratio` | len(response)/len(payload) | Normalized response size indicator |
| F6 | `has_html_tags` | 1 if `<html>\|<body>\|<div>\|<p>` in body | Generic structural indicator |
| F7 | `response_length_anomaly` | Deviation from per-type mean | Statistical anomaly detection |
| F8 | `payload_special_chars_ratio` | Ratio of `'"<>(){}[];%&\|` in payload | Payload complexity metric |

### 5.2 Model Selection Justification

| Criterion | Random Forest | Logistic Regression | SVM | Neural Network |
|-----------|:---:|:---:|:---:|:---:|
| Interpretability | ✅ High | ✅ High | ❌ Low | ❌ Low |
| Feature importance | ✅ Built-in | ❌ No | ❌ No | ❌ No |
| Small dataset handling | ✅ Robust | ✅ OK | ⚠️ Medium | ❌ Poor |
| No scaling required | ✅ Yes | ❌ No | ❌ No | ❌ No |
| Overfitting resistance | ✅ Ensemble | ⚠️ Medium | ✅ Good | ❌ Poor |
| Training speed | ✅ Fast | ✅ Fast | ⚠️ Medium | ❌ Slow |

**Selected: Random Forest** — Best balance of interpretability (critical for academic thesis), built-in feature importance, and robustness on small datasets.

**Hyperparameters:**
- `n_estimators=100` — sufficient ensemble size for stability
- `class_weight="balanced"` — handles class imbalance (more NOT_VULNERABLE than VULNERABLE)
- `max_depth=10` — prevents overfitting on small DVWA dataset
- `min_samples_leaf=2` — requires minimum support per leaf
- `random_state=42` — reproducibility

### 5.3 Training/Evaluation Methodology

```
labeled_results.json
        │
        â–¼
 ┌──────────────┐
 │   Feature    │ → 8 behavioral features per sample
 │  Extraction  │
 └──────┬───────┘
        │ (X, y)
        â–¼
 ┌──────────────┐     ┌──────────────┐
 │  Stratified  │────>│   Training   │ → RandomForest.fit(X_train, y_train)
 │  Split 80/20 │     └──────┬───────┘
 └──────┬───────┘            │
        │ X_test,y_test      ▼
        │              ┌──────────────┐
        └─────────────>│  Evaluation  │ → accuracy, F1, confusion matrix
                       └──────────────┘
                             │
 ┌──────────────┐            │
 │  5-Fold CV   │◄───────────┘ (on full X, y)
 └──────┬───────┘
        │
        â–¼
 TrainingReport + model.pkl
```

**Reported Performance on DVWA:**
- Accuracy: 100% | F1-Score: 100% | CV: 1.00 ± 0.00
- Top importances: `response_len` (55%), `vuln_type_encoded` (33.5%), `payload_len` (11.6%)

### 5.4 Limitations and Generalization Concerns

1. **DVWA-specific training:** Model is trained exclusively on DVWA data; generalization to other applications requires retraining
2. **Perfect scores warning:** 100% accuracy on DVWA indicates clear behavioral separation, but may not hold on more complex targets
3. **Feature set stability:** Behavioral features depend on HTTP response characteristics that may vary across server technologies
4. **Minimum sample requirement:** Needs ≥4 labeled samples; skipped for very small scan results

---

## 6. Database / Persistence Design

### 6.1 JSON File Schemas

**results.json (Crawler output):**
```json
{
  "target": "http://localhost:8080",
  "pages_visited": 33,
  "forms": [
    {
      "url": "http://localhost:8080/vulnerabilities/sqli/",
      "action": "http://localhost:8080/vulnerabilities/sqli/",
      "method": "GET",
      "inputs": [
        {"name": "id", "type": "text", "value": ""},
        {"name": "Submit", "type": "submit", "value": "Submit"}
      ]
    }
  ],
  "url_params": [
    {"url": "http://localhost:8080/vulnerabilities/sqli/?id=1", "params": ["id"]}
  ]
}
```

**injection_results.json (Injector output):**
```json
[
  {
    "target_url": "http://localhost:8080/vulnerabilities/sqli/",
    "method": "GET",
    "vuln_type": "sqli",
    "param": "id",
    "payload": "' OR 1=1 --",
    "status_code": 200,
    "response_len": 4523,
    "response_body": "<html>...First name: admin..."
  }
]
```

**labeled_results.json (Analyzer output):**
```json
[
  {
    "target_url": "...", "method": "GET", "vuln_type": "sqli",
    "param": "id", "payload": "' OR 1=1 --",
    "status_code": 200, "response_len": 4523, "response_body": "...",
    "label": "VULNERABLE",
    "reason": "Data dump: 5 rows returned",
    "severity": "HIGH"
  }
]
```

**model.pkl (Classifier output):**
```python
{
    "model": <sklearn.ensemble.RandomForestClassifier>,
    "mean_response_len_per_type": {0: 4523.0, 1: 3200.0, 2: 2100.0}
}
```

**all_payloads.json (PayloadGenerator output):**
```json
{
  "sqli": {"base": 13, "generated": 208, "added": 195, "payloads": ["...", "..."]},
  "xss":  {"base": 12, "generated": 150, "added": 138, "payloads": ["..."]},
  "lfi":  {"base": 11, "generated": 80,  "added": 69,  "payloads": ["..."]}
}
```

### 6.2 Results Directory Structure

```
IntelliScan/
├── results/
│   ├── results.json              # Crawler output
│   ├── injection_results.json    # Injector output
│   ├── labeled_results.json      # Analyzer output
│   ├── all_payloads.json         # PayloadGenerator output
│   └── report.pdf                # Reporter output
├── models/
│   └── model.pkl                 # Trained RF model (joblib)
└── payloads/
    ├── sqli.txt                  # 13 base SQLi payloads
    ├── xss.txt                   # 12 base XSS payloads
    └── lfi.txt                   # 11 base LFI payloads
```

## 7. Interface Design

### 7.1 CLI Interface (Click + Rich)

**Command Structure:**

```
intelliscan [OPTIONS] COMMAND [ARGS]

Options:
  -v, --verbose    Enable debug logging
  --version        Show version and exit
  --help           Show help message

Commands:
  scan               Run the full 6-module scan pipeline
  crawl              Run the crawler only
  train              Train the ML classifier on labeled data
  generate-payloads  Apply 6 mutation techniques to base payloads
```

**Scan Command Options:**

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `--target` | STRING | (required) | Target URL |
| `--auth` | STRING | None | Credentials `user:password` |
| `--report` | STRING | `report.pdf` | PDF output path |
| `--no-ml` | FLAG | False | Skip ML classifier |
| `--discord` | STRING | None | Discord webhook URL |
| `--no-tls-verify` | FLAG | False | Disable TLS verification |

**CLI Output Example:**

```
╭──────────────────────────────────────────╮
│  IntelliScan  v1.0.0                     │
│  AI-Powered Web Vulnerability Scanner    │
│  SQLi | XSS | LFI | Random Forest ML    │
╰──────────────────────────────────────────╯

        Scan Summary
┌──────────────────┬───────────────┐
│ Metric           │ Value         │
├──────────────────┼───────────────┤
│ Target           │ localhost:8080│
│ Duration         │ 45.2s         │
│ Pages crawled    │ 33            │
│ Forms found      │ 12            │
│ Injections       │ 144           │
│ Vulnerabilities  │ 38 / 144      │
│ ML Accuracy      │ 100.0%        │
│ PDF Report       │ report.pdf    │
└──────────────────┴───────────────┘

      Findings by Type
┌──────┬────────┬────────────┬──────┐
│ Type │ Tested │ Vulnerable │ Rate │
├──────┼────────┼────────────┼──────┤
│ SQLI │   52   │     13     │ 100% │
│ XSS_R│   48   │     12     │ 100% │
│ XSS_S│   48   │     12     │ 100% │
│ LFI  │   44   │      1     │  10% │
└──────┴────────┴────────────┴──────┘
```

### 7.2 Web Dashboard (Flask)

**Planned Endpoints:**

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/` | GET | Dashboard home with scan history |
| `/scan` | POST | Trigger a new scan |
| `/scan/<id>` | GET | View scan results |
| `/api/scan` | POST | REST API — start scan |
| `/api/results/<id>` | GET | REST API — fetch results JSON |

**Dashboard Visualizations (Chart.js):**
- Pie chart: vulnerability distribution by type
- Bar chart: detection rates per vuln type
- Timeline: scan history with severity trends
- Gauge: ML confidence score

### 7.3 PDF Report Layout

```
┌─────────────────────────────────────────┐
│           PAGE 1 — COVER                │
│                                         │
│     IntelliScan Security Report         │
│   AI-Powered Vulnerability Assessment   │
│                                         │
│  ┌─────────────────────────────────┐    │
│  │ Target: http://localhost:8080   │    │
│  │ Date:   2026-05-11T19:00:00    │    │
│  │ Tool:   IntelliScan v1.0       │    │
│  └─────────────────────────────────┘    │
│                                         │
│   Findings by severity                  │
│  ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐   │
│  │ HIGH │ │MEDIUM│ │ LOW  │ │ INFO │   │
│  │  26  │ │  12  │ │  0   │ │  0   │   │
│  │ RED  │ │ORANGE│ │YELLOW│ │GREEN │   │
│  └──────┘ └──────┘ └──────┘ └──────┘   │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│        PAGE 2 — EXECUTIVE SUMMARY       │
│                                         │
│  ┌──────┬──────┬──────┬──────┬──────┐   │
│  │ Type │Tested│ Vuln │ Rate │ Sev  │   │
│  ├──────┼──────┼──────┼──────┼──────┤   │
│  │ SQLI │  52  │  13  │ 100% │ HIGH │   │
│  │ XSS_R│  48  │  12  │ 100% │MEDIUM│   │
│  │ XSS_S│  48  │  12  │ 100% │ HIGH │   │
│  │ LFI  │  44  │   1  │  10% │ HIGH │   │
│  └──────┴──────┴──────┴──────┴──────┘   │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│       PAGE 3+ — DETAILED FINDINGS       │
│                                         │
│  ┌─────┐ Finding #1 — SQLI             │
│  │ HIGH│                                │
│  └─────┘                                │
│  URL:      /vulnerabilities/sqli/       │
│  Method:   GET                          │
│  Param:    id                           │
│  Payload:  ' OR 1=1 --                  │
│  Reason:   Data dump: 5 rows returned   │
│  Status:   200                          │
│  Resp:     4523 bytes                   │
│                                         │
│  (repeated for each vulnerability)      │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│       PAGE N — RECOMMENDATIONS          │
│                                         │
│  For SQLI vulnerabilities:              │
│  • Use prepared statements (PDO)        │
│  • Apply ORM frameworks                 │
│  • Deploy WAF with SQLi rules           │
│  • Input validation (whitelist)         │
│  • Least-privilege DB accounts          │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│       PAGE N+1 — METHODOLOGY            │
│                                         │
│  • Crawler: BFS + CSRF handling         │
│  • Injector: GET/POST concurrent        │
│  • Analyzer: Signature-based detection  │
│  • ML: Random Forest (100 trees)        │
│  • Payload Gen: 6 mutation techniques   │
│  • Reporter: Professional PDF           │
│                                         │
│  Disclaimer: authorized testing only    │
└─────────────────────────────────────────┘
```

### 7.4 Discord Notification Format

```
╔══════════════════════════════════════════╗
â•‘  IntelliScan: scan complete             â•‘
╠══════════════════════════════════════════╣
â•‘  Target:    localhost:8080              â•‘
â•‘  Duration:  45.2s                       â•‘
╠══════════════════════════════════════════╣
║  SQLI: 13  │  XSS_R: 12  │  LFI: 1    ║
╠══════════════════════════════════════════╣
â•‘  Color: RED (vulnerabilities found)     â•‘
â•‘  Footer: IntelliScan v1.0              â•‘
╚══════════════════════════════════════════╝
```

---

## 8. Security & Ethical Design

### 8.1 Authorization Model

IntelliScan operates under a **strict authorization-first** model:

1. **No default target:** The `--target` flag is mandatory; no automatic scanning
2. **User-Agent identification:** Every request is tagged with `IntelliScan/1.0`
3. **Credential handling:** Auth credentials are passed via CLI, never stored on disk
4. **Session isolation:** Each scan creates a new `requests.Session`, closed after scan

### 8.2 Rate Limiting and Responsible Scanning

| Control | Value | Purpose |
|---------|-------|---------|
| `RATE_LIMIT_DELAY` | 0.1s | Minimum inter-request delay |
| `MAX_CONCURRENT` | 8 | Thread pool worker limit |
| `MAX_PAGES` | 200 | Maximum pages crawled |
| `MAX_CRAWL_DEPTH` | 5 | Maximum BFS depth |
| `RESPONSE_BODY_TRUNCATE` | 8000 | Memory protection |
| `DEFAULT_TIMEOUT` | 10s | Request timeout |
| Retry backoff | 0.5s exponential | Avoids flooding on errors |

### 8.3 Data Handling

- **Scan results** may contain sensitive data (SQL dumps, file contents). Results are stored locally in `results/` with no automatic transmission
- **Model files** (`model.pkl`) contain only statistical model weights, no raw data
- **PDF reports** may contain payload/response excerpts; treat as confidential
- **Discord notifications** transmit only aggregate counts, never raw payloads

### 8.4 Legal Compliance

| Regulation | Applicability | IntelliScan Compliance |
|------------|---------------|----------------------|
| **Loi 09-08** (Morocco) | Processing scan data containing PII | Results stored locally; user responsible for data retention policy |
| **RGPD/GDPR** (EU) | Scanning EU-hosted targets | No personal data collected beyond HTTP responses; data minimization via truncation |
| **CFAA** (US) | Unauthorized computer access | Tool requires explicit `--target` and `--auth`; documentation mandates written authorization |
| **Computer Misuse Act** (UK) | Unauthorized access | Same as CFAA compliance |

---

## 9. Test Strategy

### 9.1 Unit Test Coverage Plan

| Module | Test File | Key Tests | Coverage Target |
|--------|-----------|-----------|----------------|
| Analyzer | `test_analyzer.py` | SQLi detection, XSS reflection, LFI signatures, severity mapping | >90% |
| Classifier | `test_classifier.py` | Feature extraction, training on mock data, model persistence | >85% |
| Payload Gen | `test_payload_gen.py` | Each mutation technique, deduplication, type-specific filtering | >90% |
| Reporter | `test_reporter.py` | PDF generation, section creation, sanitization | >80% |
| Crawler | (planned) | BFS traversal, form extraction, blacklist, same-domain | >85% |
| Injector | (planned) | Task building, concurrent execution, CSRF handling | >80% |

### 9.2 Integration Test Strategy

**Docker-based integration testing:**
```
┌────────────┐      HTTP       ┌──────────┐
│ IntelliScan│ ◄──────────────►│   DVWA   │
│  (pytest)  │    port 8080    │ (Docker) │
└────────────┘                 └──────────┘
```

1. Start DVWA via Docker Compose
2. Run full scan pipeline against `http://localhost:8080`
3. Verify: crawl discovers ≥4 forms, injector produces results, analyzer labels correctly
4. Verify: ML trains with accuracy ≥90%, report PDF is generated

### 9.3 Test Fixtures (`conftest.py`)

| Fixture | Content | Used By |
|---------|---------|---------|
| `sample_injection_results` | 4 mock injection entries (2 sqli, 1 xss, 1 lfi) | Analyzer, Classifier |
| `sample_labeled_results` | 4 labeled entries (2 VULNERABLE, 2 NOT_VULNERABLE) | Classifier, Reporter |

### 9.4 CI/CD Pipeline (GitHub Actions)

```yaml
# .github/workflows/ci.yml
Strategy:
  matrix:
    python-version: [3.10, 3.11, 3.12]

Steps:
  1. Checkout code
  2. Setup Python ${{ matrix.python-version }}
  3. Install dependencies (pip install -e .[dev])
  4. Run linters (ruff check, black --check, mypy)
  5. Run unit tests (pytest --cov=intelliscan --cov-report=xml)
  6. Upload coverage report
  7. Build Docker image (docker build .)
```

---

## 10. Deployment Design

### 10.1 Docker Compose Architecture

```
┌─────────────────────────────────────────────────────────┐
│              Docker Compose Network                     │
│              (intelliscan-net, bridge)                   │
│                                                         │
│  ┌─────────────────────┐   ┌─────────────────────┐     │
│  │    intelliscan       │   │       dvwa           │     │
│  │                      │   │                      │     │
│  │  Python 3.10+        │   │  Apache + PHP        │     │
│  │  Flask :5000  ◄──────┼───┼──  :80 (ext :8080)   │     │
│  │  CLI (Click)         │   │  MySQL embedded      │     │
│  │                      │   │                      │     │
│  │  Volumes:            │   │  Image:              │     │
│  │   ./results:/app/res │   │  vulnerables/        │     │
│  │   ./models:/app/mod  │   │  web-dvwa:latest     │     │
│  └─────────────────────┘   └─────────────────────┘     │
│         :5000                       :8080                │
└─────────┬───────────────────────────┬───────────────────┘
          │                           │
      Host port 5000             Host port 8080
```

### 10.2 Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `INTELLISCAN_TIMEOUT` | `10` | HTTP request timeout (seconds) |
| `INTELLISCAN_RESULTS` | `./results` | Results directory path |

### 10.3 Volume Mounts

| Host Path | Container Path | Purpose |
|-----------|---------------|---------|
| `./results` | `/app/results` | Persist scan results between container restarts |
| `./models` | `/app/models` | Persist trained ML models |

### 10.4 Scaling Considerations

- **Horizontal:** v3.0 roadmap includes Celery + Redis distributed scanning
- **Vertical:** `MAX_CONCURRENT` adjustable from 1 to 32 workers
- **Storage:** JSON files are stateless; multiple scan runs can coexist with unique filenames

---

## 11. Technology Choices Justification

### 11.1 Python 3.10+

| Reason | Detail |
|--------|--------|
| **Type hints** | Full `PEP 604` union syntax (`str \| None`), `PEP 585` generics |
| **Match statement** | `PEP 634` structural pattern matching (future use) |
| **Ecosystem** | scikit-learn, requests, BeautifulSoup — mature, well-documented |
| **Academic context** | Most taught language in cybersecurity curricula |

### 11.2 Random Forest (scikit-learn)

| Reason | Detail |
|--------|--------|
| **Interpretability** | `feature_importances_` enables academic analysis |
| **No scaling required** | Tree-based models handle heterogeneous features natively |
| **Ensemble robustness** | 100 trees reduce variance vs single decision tree |
| **Balanced class weights** | Built-in handling of class imbalance |
| **Reproducibility** | `random_state=42` ensures deterministic results |

### 11.3 fpdf2

| Reason | Detail |
|--------|--------|
| **Pure Python** | No system dependencies (unlike ReportLab, WeasyPrint) |
| **Lightweight** | ~200KB package, no C extensions |
| **Sufficient features** | Tables, colored cells, multi-cell, header/footer hooks |
| **License** | LGPL — compatible with MIT project license |

### 11.4 Flask

| Reason | Detail |
|--------|--------|
| **Lightweight** | Minimal framework, ideal for prototype dashboard |
| **Familiar** | Widely used in academic projects |
| **REST-friendly** | Easy API endpoint creation for future integrations |
| **Jinja2 templates** | Built-in templating for dashboard views |

### 11.5 Click + Rich

| Reason | Detail |
|--------|--------|
| **Click** | Decorator-based CLI definition, automatic help generation, type validation |
| **Rich** | Colored output, tables, progress bars, tracebacks — professional DX |
| **Combination** | `RichHandler` integrates logging with Rich console output |

---

## 12. Limitations & Future Work

### 12.1 Current Limitations

| # | Limitation | Impact | Mitigation |
|---|-----------|--------|------------|
| 1 | **LFI detection: ~10% on DVWA** | PHP `realpath()` in Docker blocks traversal | Higher rates on production configs; expand LFI signatures |
| 2 | **ML trained on DVWA only** | Model may not generalize to other apps | Retrain on diverse datasets; transfer learning in v2.0 |
| 3 | **Hardcoded DVWA_FORMS** | Fallback forms specific to DVWA endpoints | Crawler dynamically discovers forms in real scans |
| 4 | **No blind SQLi** | Time-based and boolean-based SQLi not detected | Planned for v1.1 |
| 5 | **No DOM XSS** | Client-side JavaScript execution not analyzed | Selenium-based detection in v2.0 |
| 6 | **Single-machine only** | No distributed scanning capability | Celery + Redis in v3.0 |

### 12.2 Roadmap

#### v1.1 — Extended Detection (Near-term)
- CSRF detection module
- Command Injection detection
- SSRF detection
- GraphQL endpoint support
- OAuth2/JWT authentication
- Blind SQLi (time-based inference)

#### v2.0 — Advanced ML (Medium-term)
- LSTM sequence classifier for payload/response analysis
- Active learning loop: analyst labels → model retraining
- DOM XSS detection via Selenium headless browser
- CI/CD plugins: GitHub Action, Jenkins plugin
- Multi-target scanning with priority queue

#### v3.0 — Enterprise Scale (Long-term)
- Distributed scanning with Celery + Redis task queue
- CVE knowledge base integration (NVD, MITRE)
- Auto-remediation suggestions via LLM (GPT-4 / local models)
- Web-based management console with user roles
- Compliance report templates (PCI-DSS, SOC 2)

---

## Appendix A — Generating Architecture Diagrams

All diagrams referenced in this document can be generated as high-resolution PNG files using the provided script.

**Prerequisites:**
```bash
pip install matplotlib graphviz
```

**Additionally install the Graphviz system package:**
- Windows: `winget install graphviz` or download from https://graphviz.org/download/
- Linux: `sudo apt install graphviz`
- macOS: `brew install graphviz`

**Generate all diagrams:**
```bash
python docs/generate_diagrams.py
```

**Output files in `docs/images/`:**

| File | Diagram |
|------|---------|
| `01_pipeline_diagram.png` | 6-module sequential pipeline |
| `02_dependency_graph.png` | Module dependency graph |
| `03_data_flow_diagram.png` | Data flow between modules |
| `04_ml_feature_importance.png` | RF feature importance bar chart |
| `05_class_diagram.png` | Class/component UML diagram |
| `06_deployment_diagram.png` | Docker Compose architecture |
| `07_ml_training_flow.png` | ML training & evaluation pipeline |
| `08_mutation_pipeline.png` | Payload mutation techniques |

If Graphviz is not installed, DOT source files (`.dot`) are saved for manual rendering.

---

## Appendix B — References

1. OWASP Top 10 (2021). https://owasp.org/Top10/
2. Breiman, L. (2001). Random forests. *Machine Learning*, 45(1), 5-32.
3. DVWA — Damn Vulnerable Web Application. https://github.com/digininja/DVWA
4. scikit-learn documentation. https://scikit-learn.org/stable/
5. fpdf2 documentation. https://py-pdf.github.io/fpdf2/
6. Click documentation. https://click.palletsprojects.com/
7. Rich documentation. https://rich.readthedocs.io/

---

*Document generated for the Master's thesis (PFE) at Ibn Tofail University, Kenitra, Morocco.*  
*Supervisor: Pr. Youssef FAKHRI — Academic year 2025–2026.*