docs/CONCEPTION_DESIGN_DOCUMENT.md

# IntelliScan — Conception & Design Document

Project: IntelliScan — AI-Powered Web Vulnerability Scanner
Version: 1.0.0
Author: SABKARI Mohamed
Institution: Ibn Tofail University, Kenitra, Morocco
Academic Context: Master's Thesis (PFE) — Cybersecurity & AI, 2025–2026
Supervisor: Pr. Youssef FAKHRI
License: MIT
Date: May 2026

---

Table of Contents

1. Project Overview
2. Functional Requirements
3. System Architecture
4. Detailed Module Design
5. ML Classifier Design
6. Database / Persistence Design
7. Interface Design
8. Security & Ethical Design
9. Test Strategy
10. Deployment Design
11. Technology Choices Justification
12. Limitations & Future Work

---

1. Project Overview

1.1 Problem Statement

Web application vulnerabilities remain one of the most exploited attack vectors in cybersecurity. According to the OWASP Top 10 (2021), injection flaws (SQLi), Cross-Site Scripting (XSS), and insecure file handling (LFI) consistently rank among the most critical threats. Existing vulnerability scanners present several limitations:

Scanner	Limitation
OWASP ZAP	Complex UI, steep learning curve, no ML-based detection
Burp Suite	Commercial license required for advanced features
Nikto	Signature-only, no adaptive classification
SQLMap	Single vulnerability type (SQLi only)
w3af	Discontinued maintenance, outdated engine

IntelliScan addresses these gaps by combining signature-based detection with machine learning classification in a unified, open-source, modular pipeline. The dual-approach design provides both the reliability of known-pattern matching and the adaptability of behavioral ML classification.

1.2 Objectives and Scope

Primary Objectives:

1. Automate authenticated crawling and vulnerability detection for SQLi, XSS, and LFI
2. Provide a dual detection engine (signature + Random Forest ML) to improve detection confidence
3. Generate professional PDF reports suitable for stakeholder communication
4. Offer a modular, extensible architecture enabling future vulnerability type additions

Scope Boundaries:

• In scope: SQLi, reflected XSS, stored XSS, LFI on HTTP/HTTPS targets
• Out of scope (v1.0): CSRF, SSRF, command injection, blind SQLi, DOM XSS, API fuzzing

1.3 Target Users

Actor	Profile	Use Case
Penetration Tester	Security professional with CLI experience	Authorized vulnerability assessments
Developer	Web developer integrating security into CI/CD	Pre-deployment security checks
Academic Researcher	Student/professor studying ML-based detection	Benchmarking detection algorithms
Security Auditor	Compliance officer reviewing application security	Generating audit evidence (PDF reports)

1.4 Ethical and Legal Framework

IntelliScan is designed exclusively for authorized security testing. The tool enforces:

• Explicit consent requirement: Users must have written authorization before scanning
• Identification header: All HTTP requests include User-Agent: IntelliScan/1.0
• Rate limiting: 0.1s delay between requests to minimize target impact
• Scope restriction: Crawler enforces same-domain policy and blacklists sensitive paths

Legal compliance:

• Morocco — Loi 09-08: Data protection for scan results containing PII
• EU — RGPD/GDPR: Applicable when scanning EU-hosted targets
• US — CFAA: Unauthorized access prohibition; IntelliScan requires explicit authorization
• Responsible Disclosure: Tool documentation recommends coordinated disclosure practices

---

2. Functional Requirements

2.1 Use Cases

┌──────────────────────────────────────────────────────────────┐
│                    IntelliScan System                        │
│                                                              │
│  ┌─────────┐    ┌──────────────────────────────────────┐     │
│  │ Security │───>│ UC-01: Run Full Scan Pipeline       │     │
│  │ Analyst  │    └──────────────────────────────────────┘     │
│  │          │    ┌──────────────────────────────────────┐     │
│  │          │───>│ UC-02: Crawl Target Only             │     │
│  │          │    └──────────────────────────────────────┘     │
│  │          │    ┌──────────────────────────────────────┐     │
│  │          │───>│ UC-03: Train ML Classifier           │     │
│  │          │    └──────────────────────────────────────┘     │
│  │          │    ┌──────────────────────────────────────┐     │
│  │          │───>│ UC-04: Generate Payload Variants     │     │
│  │          │    └──────────────────────────────────────┘     │
│  │          │    ┌──────────────────────────────────────┐     │
│  │          │───>│ UC-05: View PDF Report               │     │
│  └─────────┘    └──────────────────────────────────────┘     │
│                                                              │
│  ┌─────────┐    ┌──────────────────────────────────────┐     │
│  │ CI/CD   │───>│ UC-06: Automated Scan (CLI)          │     │
│  │ Pipeline│    └──────────────────────────────────────┘     │
│  └─────────┘                                                 │
│                                                              │
│  ┌─────────┐    ┌──────────────────────────────────────┐     │
│  │ Discord │<───│ UC-07: Receive Scan Notification     │     │
│  │ Channel │    └──────────────────────────────────────┘     │
│  └─────────┘                                                 │
└──────────────────────────────────────────────────────────────┘

2.2 Functional Requirements per Module

Module 1 — Crawler

ID	Requirement	Priority
FR-CRAWL-01	Authenticate via POST /login.php with CSRF token extraction	Must
FR-CRAWL-02	Set DVWA security level to "low" automatically	Must
FR-CRAWL-03	BFS traversal with configurable MAX_DEPTH (default=5)	Must
FR-CRAWL-04	Limit to MAX_PAGES (default=200) to prevent infinite crawling	Must
FR-CRAWL-05	Extract HTML forms with action, method, input names/types/values	Must
FR-CRAWL-06	Extract URL query parameters from discovered pages	Must
FR-CRAWL-07	Enforce same-domain policy (no off-site crawling)	Must
FR-CRAWL-08	Blacklist sensitive paths (/logout.php, /setup.php, /.git/, /.env)	Must
FR-CRAWL-09	Output results.json with structured crawl data	Must

Module 2 — Injector

ID	Requirement	Priority
FR-INJ-01	Load payloads from payloads/ directory (sqli.txt, xss.txt, lfi.txt)	Must
FR-INJ-02	Support GET and POST injection methods	Must
FR-INJ-03	Concurrent injection via ThreadPoolExecutor (MAX_CONCURRENT=8)	Must
FR-INJ-04	Auto-fetch CSRF token for POST forms (user_token)	Must
FR-INJ-05	Capture status_code, response_len, response_body (truncated 8000 chars)	Must
FR-INJ-06	Support DVWA_FORMS fallback definitions for 4 endpoints	Should
FR-INJ-07	Output injection_results.json	Must

Module 3 — Analyzer

ID	Requirement	Priority
FR-ANAL-01	Detect SQLi via "first name:" count and SQL_ERROR_PATTERNS (8 patterns)	Must
FR-ANAL-02	Detect XSS via payload reflection check and XSS_DANGEROUS_PATTERNS	Must
FR-ANAL-03	Detect LFI via LFI_SIGNATURES (root:x:0:0, /bin/bash, etc.)	Must
FR-ANAL-04	Assign severity: sqli=HIGH, xss_r=MEDIUM, xss_s=HIGH, lfi=HIGH	Must
FR-ANAL-05	Label each result as VULNERABLE or NOT_VULNERABLE with reason	Must
FR-ANAL-06	Output labeled_results.json	Must

Module 4 — ML Classifier

ID	Requirement	Priority
FR-ML-01	Extract 8 behavioral features (no data leakage)	Must
FR-ML-02	Train RandomForestClassifier (100 trees, balanced, max_depth=10)	Must
FR-ML-03	80/20 stratified train-test split	Must
FR-ML-04	5-fold cross-validation with F1-weighted scoring	Must
FR-ML-05	Persist model + mean_response_len_per_type via joblib	Must
FR-ML-06	Require minimum 4 samples to train	Must
FR-ML-07	Produce TrainingReport with metrics and feature importances	Must

Module 5 — Payload Generator

ID	Requirement	Priority
FR-GEN-01	Case alternation mutation (not for LFI)	Must
FR-GEN-02	SQL comment insertion (sqli only)	Must
FR-GEN-03	Quote substitution ' ↔ " (not for LFI)	Must
FR-GEN-04	URL encoding (urllib.parse.quote)	Must
FR-GEN-05	Double URL encoding	Must
FR-GEN-06	Whitespace variation (double space, tab, /**/)	Must
FR-GEN-07	Deduplicate generated variants	Must

Module 6 — Reporter

ID	Requirement	Priority
FR-REP-01	Cover page with target, date, severity summary boxes	Must
FR-REP-02	Executive summary with statistics table by vuln type	Must
FR-REP-03	Detailed findings with severity badge, URL, payload, reason	Must
FR-REP-04	Remediation recommendations (5 per vuln type)	Must
FR-REP-05	Methodology & disclaimer section	Must
FR-REP-06	Output report.pdf using fpdf2	Must

2.3 Non-Functional Requirements

ID	Category	Requirement	Target
NFR-01	Performance	Complete full scan of DVWA in < 5 minutes	< 300s
NFR-02	Performance	Concurrent injection with 8 workers	4-8x speedup
NFR-03	Reliability	Retry failed HTTP requests (3 retries, exponential backoff)	99% delivery
NFR-04	Security	Rate limit requests to 0.1s minimum interval	Responsible scanning
NFR-05	Portability	Run on Python 3.10+ across Windows, Linux, macOS	Cross-platform
NFR-06	Portability	Docker Compose deployment with DVWA test target	One-command setup
NFR-07	Maintainability	Modular pipeline architecture (6 independent modules)	Low coupling
NFR-08	Maintainability	>80% unit test coverage	pytest-cov
NFR-09	Usability	Rich CLI output with colored tables and progress	Developer UX
NFR-10	Extensibility	JSON intermediate files enable module replacement	Plugin-ready

3. System Architecture

3.1 High-Level Pipeline Diagram

 ┌─────────────┐     ┌─────────────┐     ┌─────────────┐
 │   TARGET     │     │  PAYLOADS   │     │   CONFIG     │
 │  Web App     │     │ sqli.txt    │     │  config.py   │
 │  (DVWA)      │     │ xss.txt     │     │  Constants   │
 └──────┬───────┘     │ lfi.txt     │     └──────┬───────┘
        │             └──────┬──────┘            │
        ▼                    │                   │
 ┌──────────────┐            │         ┌─────────▼────────┐
 │  MODULE 1    │            │         │   HttpClient     │
 │  Crawler     │◄───────────┼─────────│   (Session,      │
 │  BFS + Auth  │            │         │    Retry, CSRF)  │
 └──────┬───────┘            │         └──────────────────┘
        │                    │
        ▼ results.json       │
 ┌──────────────┐            │
 │  MODULE 2    │◄───────────┘
 │  Injector    │
 │  GET / POST  │
 └──────┬───────┘
        │
        â–¼ injection_results.json
 ┌──────────────┐
 │  MODULE 3    │
 │  Analyzer    │
 │  Signatures  │
 └──────┬───────┘
        │
        â–¼ labeled_results.json
   ┌────┴─────────────────┐
   │                      │
   â–¼                      â–¼
 ┌──────────────┐  ┌──────────────┐
 │  MODULE 4    │  │  MODULE 6    │
 │  ML Class.   │  │  Reporter    │
 │  Rand Forest │  │  PDF (fpdf2) │
 └──────┬───────┘  └──────┬───────┘
        │                 │
        â–¼                 â–¼
   model.pkl         report.pdf

 ┌──────────────┐
 │  MODULE 5    │ (independent — can run standalone)
 │  Payload Gen │
 │  6 Mutations │
 └──────┬───────┘
        │
        â–¼
  all_payloads.json

Note: Generate high-resolution PNG versions with python docs/generate_diagrams.py

3.2 Module Dependency Graph

                    ┌─────────────────┐
                    │   __main__.py   │
                    │   CLI (Click)   │
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │    core.py      │
                    │  IntelliScan    │
                    └────────┬────────┘
                             │
         ┌───────┬───────┬───┼───┬───────┬───────┐
         ▼       ▼       ▼   │   ▼       ▼       ▼
     crawler  injector  anal │  classif  pgen  reporter
       .py     .py      .py │   .py     .py    .py
         │       │       │   │   │       │       │
         ▼       ▼       │   │   │       │       │
      http_client.py     │   │   │       │       │
                         â–¼   â–¼   â–¼       â–¼       â–¼
                      ┌──────────────────────────┐
                      │       config.py          │
                      │  (All constants, Final)  │
                      └──────────────────────────┘

3.3 Component Interaction Description

The IntelliScan orchestrator (core.py) implements the Pipeline design pattern:

1. Crawler authenticates and explores the target via BFS, producing a site map
2. Injector reads discovered forms/params and delivers payloads concurrently
3. Analyzer applies signature-based heuristics to label each injection result
4. ML Classifier trains a Random Forest on the labeled data for secondary classification
5. Payload Generator produces WAF-bypass variants (runs independently)
6. Reporter compiles all findings into a professional PDF report

Key design principles:

• Loose coupling: Modules communicate exclusively via JSON files
• Single responsibility: Each module has one clearly defined role
• Fail-safe orchestration: ML training is skipped if <4 samples; individual module failures don't crash the pipeline

---

4. Detailed Module Design

4.1 Module 1 — Crawler (crawler.py)

Purpose: Authenticated BFS exploration of the target web application to discover all injectable forms and URL parameters.

Input/Output:

Direction	Data	Format
Input	Target URL, optional credentials (user:password)	CLI arguments
Output	Site map with forms and URL params	results.json

Key Algorithms:

• BFS traversal using collections.deque with (url, depth) tuples
• CSRF token extraction via BeautifulSoup parsing of input[name=user_token]
• Form extraction parsing <form>, <input>, <textarea>, <select> elements
• Same-domain enforcement via urlparse().netloc comparison

Key Classes:

Class	Fields	Role
Form	url, action, method, inputs	Represents a discovered HTML form
UrlParam	url, params	Represents a URL with query parameters
CrawlResult	target, pages_visited, forms, url_params	Aggregated crawl output
Crawler	target, auth, max_pages, max_depth, http	Main crawler engine

Error Handling:

• HTTP errors during page fetching are logged and skipped (BFS continues)
• Authentication failure raises RuntimeError with clear message
• DVWA security configuration failure is silently ignored (target may not be DVWA)

Testability:

• Mock HttpClient to provide canned HTML responses
• Test BFS depth limiting, blacklist enforcement, form extraction independently

---

4.2 Module 2 — Injector (injector.py)

Purpose: Inject attack payloads into all discovered forms and URL parameters, capturing HTTP responses for analysis.

Input/Output:

Direction	Data	Format
Input	Crawl results, payload files	results.json, payloads/*.txt
Output	Injection results with responses	injection_results.json

Key Algorithms:

• Task generation: Cartesian product of (vuln_type × form × payload)
• Concurrent execution: ThreadPoolExecutor with MAX_CONCURRENT=8 workers
• CSRF handling: Auto-fetches fresh token for each POST injection

Payload Counts (base):

File	Payloads	Examples
sqli.txt	13	' OR 1=1 --, ' UNION SELECT, '; DROP TABLE
xss.txt	12	<script>alert(1)</script>, <img onerror=...>
lfi.txt	11	../../../../etc/passwd, php://filter/...

DVWA Target Endpoints:

Vuln Type	URL	Method	Parameters
sqli	/vulnerabilities/sqli/	GET	id
xss_r	/vulnerabilities/xss_r/	GET	name
xss_s	/vulnerabilities/xss_s/	POST	txtName, mtxMessage
lfi	/vulnerabilities/fi/	GET	page

Error Handling:

• Individual injection failures return None (skipped in results)
• Response body truncated to 8000 characters to manage memory

---

4.3 Module 3 — Analyzer (analyzer.py)

Purpose: Signature-based labeling of each injection result as VULNERABLE or NOT_VULNERABLE.

Detection Logic per Type:

SQLi Detection:

IF body.count("first name:") >= 1 → VULNERABLE (data dump)
ELSE IF any SQL_ERROR_PATTERNS in body → VULNERABLE (SQL error)
ELSE → NOT_VULNERABLE

SQL_ERROR_PATTERNS (8 signatures):

1. you have an error in your sql syntax
2. mysql_fetch_array()
3. mysql_num_rows()
4. supplied argument is not a valid mysql
5. warning: mysql_
6. unclosed quotation mark
7. ora-01756
8. syntax error in string in query expression

XSS Detection:

IF payload.lower() in response_body → VULNERABLE (reflection)
ELSE IF any XSS_DANGEROUS_PATTERNS reflected → VULNERABLE
ELSE → NOT_VULNERABLE

LFI Detection:

IF any LFI_SIGNATURES in body → VULNERABLE
ELSE → NOT_VULNERABLE

Severity Assignment:

Vuln Type	Severity	Rationale
sqli	HIGH	Direct database access, data exfiltration
xss_r	MEDIUM	Requires victim interaction (reflected)
xss_s	HIGH	Persistent, affects all users
lfi	HIGH	Server-side file disclosure, potential RCE

---

4.4 Module 4 — ML Classifier (classifier.py)

(Detailed in Section 5)

---

4.5 Module 5 — Payload Generator (payload_gen.py)

Purpose: Produce WAF-bypass variants of base payloads via 6 mutation techniques to increase test coverage.

Mutation Techniques:

#	Technique	Example	Applicability
1	Case alternation	OR → oR, Or	sqli, xss (not lfi)
2	SQL comment insertion	OR 1=1 → OR/**/1=1	sqli only
3	Quote substitution	' ↔ "	sqli, xss (not lfi)
4	URL encoding	' → %27	all
5	Double URL encoding	' → %2527	all
6	Whitespace variation	space → tab, /**/	all

Coverage Expansion:

Type	Base	Generated	Expansion
sqli	13	~208	+530%
xss	12	~150	+530%
lfi	11	~80	+300%

---

4.6 Module 6 — Reporter (reporter.py)

Purpose: Generate a professional 5-section PDF security report using fpdf2.

Report Structure:

Section	Content	Page(s)
Cover	Target, date, tool name, severity summary boxes	1
Executive Summary	Statistics table by vuln type, detection rates	1
Detailed Findings	Per-vulnerability cards with severity badge	1-N
Recommendations	5 remediation steps per vuln type	1-2
Methodology	Module descriptions, disclaimer	1

Severity Color Coding:

Severity	RGB Color	Visual
HIGH	(192, 57, 43)	Red
MEDIUM	(243, 156, 18)	Orange
LOW	(241, 196, 15)	Yellow
INFO	(39, 174, 96)	Green

---

5. ML Classifier Design

5.1 Feature Engineering Rationale

The classifier uses 8 purely behavioral features extracted from HTTP responses. This design was chosen after a critical revision that identified data leakage in the previous feature set.

Previous features (REMOVED — data leakage):

• has_alert — directly mirrors XSS analyzer decision
• has_first_name — directly mirrors SQLi analyzer decision
• payload_in_response — directly mirrors XSS reflection check

Current 8 behavioral features:

#	Feature	Description	Rationale
F1	response_len	Body length in bytes	Vulnerable responses often contain additional data (SQL dumps, file contents)
F2	status_code	HTTP status code	Error codes may indicate injection success/failure
F3	payload_len	Length of injected payload	Complex payloads may have different success rates
F4	vuln_type_encoded	sqli=0, xss=1, lfi=2	Different vuln types have different response patterns
F5	response_to_payload_ratio	len(response)/len(payload)	Normalized response size indicator
F6	has_html_tags	1 if <html>|<body>|<div>|<p> in body	Generic structural indicator
F7	response_length_anomaly	Deviation from per-type mean	Statistical anomaly detection
F8	payload_special_chars_ratio	Ratio of '"<>(){}[];%&| in payload	Payload complexity metric

5.2 Model Selection Justification

Criterion	Random Forest	Logistic Regression	SVM	Neural Network
Interpretability	✅ High	✅ High	❌ Low	❌ Low
Feature importance	✅ Built-in	❌ No	❌ No	❌ No
Small dataset handling	✅ Robust	✅ OK	⚠️ Medium	❌ Poor
No scaling required	✅ Yes	❌ No	❌ No	❌ No
Overfitting resistance	✅ Ensemble	⚠️ Medium	✅ Good	❌ Poor
Training speed	✅ Fast	✅ Fast	⚠️ Medium	❌ Slow

Selected: Random Forest — Best balance of interpretability (critical for academic thesis), built-in feature importance, and robustness on small datasets.

Hyperparameters:

• n_estimators=100 — sufficient ensemble size for stability
• class_weight="balanced" — handles class imbalance (more NOT_VULNERABLE than VULNERABLE)
• max_depth=10 — prevents overfitting on small DVWA dataset
• min_samples_leaf=2 — requires minimum support per leaf
• random_state=42 — reproducibility

5.3 Training/Evaluation Methodology

labeled_results.json
        │
        â–¼
 ┌──────────────┐
 │   Feature    │ → 8 behavioral features per sample
 │  Extraction  │
 └──────┬───────┘
        │ (X, y)
        â–¼
 ┌──────────────┐     ┌──────────────┐
 │  Stratified  │────>│   Training   │ → RandomForest.fit(X_train, y_train)
 │  Split 80/20 │     └──────┬───────┘
 └──────┬───────┘            │
        │ X_test,y_test      ▼
        │              ┌──────────────┐
        └─────────────>│  Evaluation  │ → accuracy, F1, confusion matrix
                       └──────────────┘
                             │
 ┌──────────────┐            │
 │  5-Fold CV   │◄───────────┘ (on full X, y)
 └──────┬───────┘
        │
        â–¼
 TrainingReport + model.pkl

Reported Performance on DVWA:

• Accuracy: 100% | F1-Score: 100% | CV: 1.00 ± 0.00
• Top importances: response_len (55%), vuln_type_encoded (33.5%), payload_len (11.6%)

5.4 Limitations and Generalization Concerns

1. DVWA-specific training: Model is trained exclusively on DVWA data; generalization to other applications requires retraining
2. Perfect scores warning: 100% accuracy on DVWA indicates clear behavioral separation, but may not hold on more complex targets
3. Feature set stability: Behavioral features depend on HTTP response characteristics that may vary across server technologies
4. Minimum sample requirement: Needs ≥4 labeled samples; skipped for very small scan results

---

6. Database / Persistence Design

6.1 JSON File Schemas

results.json (Crawler output):

{
  "target": "http://localhost:8080",
  "pages_visited": 33,
  "forms": [
    {
      "url": "http://localhost:8080/vulnerabilities/sqli/",
      "action": "http://localhost:8080/vulnerabilities/sqli/",
      "method": "GET",
      "inputs": [
        {"name": "id", "type": "text", "value": ""},
        {"name": "Submit", "type": "submit", "value": "Submit"}
      ]
    }
  ],
  "url_params": [
    {"url": "http://localhost:8080/vulnerabilities/sqli/?id=1", "params": ["id"]}
  ]
}

injection_results.json (Injector output):

[
  {
    "target_url": "http://localhost:8080/vulnerabilities/sqli/",
    "method": "GET",
    "vuln_type": "sqli",
    "param": "id",
    "payload": "' OR 1=1 --",
    "status_code": 200,
    "response_len": 4523,
    "response_body": "<html>...First name: admin..."
  }
]

labeled_results.json (Analyzer output):

[
  {
    "target_url": "...", "method": "GET", "vuln_type": "sqli",
    "param": "id", "payload": "' OR 1=1 --",
    "status_code": 200, "response_len": 4523, "response_body": "...",
    "label": "VULNERABLE",
    "reason": "Data dump: 5 rows returned",
    "severity": "HIGH"
  }
]

model.pkl (Classifier output):

{
    "model": <sklearn.ensemble.RandomForestClassifier>,
    "mean_response_len_per_type": {0: 4523.0, 1: 3200.0, 2: 2100.0}
}

all_payloads.json (PayloadGenerator output):

{
  "sqli": {"base": 13, "generated": 208, "added": 195, "payloads": ["...", "..."]},
  "xss":  {"base": 12, "generated": 150, "added": 138, "payloads": ["..."]},
  "lfi":  {"base": 11, "generated": 80,  "added": 69,  "payloads": ["..."]}
}

6.2 Results Directory Structure

IntelliScan/
├── results/
│   ├── results.json              # Crawler output
│   ├── injection_results.json    # Injector output
│   ├── labeled_results.json      # Analyzer output
│   ├── all_payloads.json         # PayloadGenerator output
│   └── report.pdf                # Reporter output
├── models/
│   └── model.pkl                 # Trained RF model (joblib)
└── payloads/
    ├── sqli.txt                  # 13 base SQLi payloads
    ├── xss.txt                   # 12 base XSS payloads
    └── lfi.txt                   # 11 base LFI payloads

7. Interface Design

7.1 CLI Interface (Click + Rich)

Command Structure:

intelliscan [OPTIONS] COMMAND [ARGS]

Options:
  -v, --verbose    Enable debug logging
  --version        Show version and exit
  --help           Show help message

Commands:
  scan               Run the full 6-module scan pipeline
  crawl              Run the crawler only
  train              Train the ML classifier on labeled data
  generate-payloads  Apply 6 mutation techniques to base payloads

Scan Command Options:

Option	Type	Default	Description
--target	STRING	(required)	Target URL
--auth	STRING	None	Credentials user:password
--report	STRING	report.pdf	PDF output path
--no-ml	FLAG	False	Skip ML classifier
--discord	STRING	None	Discord webhook URL
--no-tls-verify	FLAG	False	Disable TLS verification

CLI Output Example:

╭──────────────────────────────────────────╮
│  IntelliScan  v1.0.0                     │
│  AI-Powered Web Vulnerability Scanner    │
│  SQLi | XSS | LFI | Random Forest ML    │
╰──────────────────────────────────────────╯

        Scan Summary
┌──────────────────┬───────────────┐
│ Metric           │ Value         │
├──────────────────┼───────────────┤
│ Target           │ localhost:8080│
│ Duration         │ 45.2s         │
│ Pages crawled    │ 33            │
│ Forms found      │ 12            │
│ Injections       │ 144           │
│ Vulnerabilities  │ 38 / 144      │
│ ML Accuracy      │ 100.0%        │
│ PDF Report       │ report.pdf    │
└──────────────────┴───────────────┘

      Findings by Type
┌──────┬────────┬────────────┬──────┐
│ Type │ Tested │ Vulnerable │ Rate │
├──────┼────────┼────────────┼──────┤
│ SQLI │   52   │     13     │ 100% │
│ XSS_R│   48   │     12     │ 100% │
│ XSS_S│   48   │     12     │ 100% │
│ LFI  │   44   │      1     │  10% │
└──────┴────────┴────────────┴──────┘

7.2 Web Dashboard (Flask)

Planned Endpoints:

Endpoint	Method	Description
/	GET	Dashboard home with scan history
/scan	POST	Trigger a new scan
/scan/<id>	GET	View scan results
/api/scan	POST	REST API — start scan
/api/results/<id>	GET	REST API — fetch results JSON

Dashboard Visualizations (Chart.js):

• Pie chart: vulnerability distribution by type
• Bar chart: detection rates per vuln type
• Timeline: scan history with severity trends
• Gauge: ML confidence score

7.3 PDF Report Layout

┌─────────────────────────────────────────┐
│           PAGE 1 — COVER                │
│                                         │
│     IntelliScan Security Report         │
│   AI-Powered Vulnerability Assessment   │
│                                         │
│  ┌─────────────────────────────────┐    │
│  │ Target: http://localhost:8080   │    │
│  │ Date:   2026-05-11T19:00:00    │    │
│  │ Tool:   IntelliScan v1.0       │    │
│  └─────────────────────────────────┘    │
│                                         │
│   Findings by severity                  │
│  ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐   │
│  │ HIGH │ │MEDIUM│ │ LOW  │ │ INFO │   │
│  │  26  │ │  12  │ │  0   │ │  0   │   │
│  │ RED  │ │ORANGE│ │YELLOW│ │GREEN │   │
│  └──────┘ └──────┘ └──────┘ └──────┘   │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│        PAGE 2 — EXECUTIVE SUMMARY       │
│                                         │
│  ┌──────┬──────┬──────┬──────┬──────┐   │
│  │ Type │Tested│ Vuln │ Rate │ Sev  │   │
│  ├──────┼──────┼──────┼──────┼──────┤   │
│  │ SQLI │  52  │  13  │ 100% │ HIGH │   │
│  │ XSS_R│  48  │  12  │ 100% │MEDIUM│   │
│  │ XSS_S│  48  │  12  │ 100% │ HIGH │   │
│  │ LFI  │  44  │   1  │  10% │ HIGH │   │
│  └──────┴──────┴──────┴──────┴──────┘   │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│       PAGE 3+ — DETAILED FINDINGS       │
│                                         │
│  ┌─────┐ Finding #1 — SQLI             │
│  │ HIGH│                                │
│  └─────┘                                │
│  URL:      /vulnerabilities/sqli/       │
│  Method:   GET                          │
│  Param:    id                           │
│  Payload:  ' OR 1=1 --                  │
│  Reason:   Data dump: 5 rows returned   │
│  Status:   200                          │
│  Resp:     4523 bytes                   │
│                                         │
│  (repeated for each vulnerability)      │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│       PAGE N — RECOMMENDATIONS          │
│                                         │
│  For SQLI vulnerabilities:              │
│  • Use prepared statements (PDO)        │
│  • Apply ORM frameworks                 │
│  • Deploy WAF with SQLi rules           │
│  • Input validation (whitelist)         │
│  • Least-privilege DB accounts          │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│       PAGE N+1 — METHODOLOGY            │
│                                         │
│  • Crawler: BFS + CSRF handling         │
│  • Injector: GET/POST concurrent        │
│  • Analyzer: Signature-based detection  │
│  • ML: Random Forest (100 trees)        │
│  • Payload Gen: 6 mutation techniques   │
│  • Reporter: Professional PDF           │
│                                         │
│  Disclaimer: authorized testing only    │
└─────────────────────────────────────────┘

7.4 Discord Notification Format

╔══════════════════════════════════════════╗
â•‘  IntelliScan: scan complete             â•‘
╠══════════════════════════════════════════╣
â•‘  Target:    localhost:8080              â•‘
â•‘  Duration:  45.2s                       â•‘
╠══════════════════════════════════════════╣
║  SQLI: 13  │  XSS_R: 12  │  LFI: 1    ║
╠══════════════════════════════════════════╣
â•‘  Color: RED (vulnerabilities found)     â•‘
â•‘  Footer: IntelliScan v1.0              â•‘
╚══════════════════════════════════════════╝

---

8. Security & Ethical Design

8.1 Authorization Model

IntelliScan operates under a strict authorization-first model:

1. No default target: The --target flag is mandatory; no automatic scanning
2. User-Agent identification: Every request is tagged with IntelliScan/1.0
3. Credential handling: Auth credentials are passed via CLI, never stored on disk
4. Session isolation: Each scan creates a new requests.Session, closed after scan

8.2 Rate Limiting and Responsible Scanning

Control	Value	Purpose
RATE_LIMIT_DELAY	0.1s	Minimum inter-request delay
MAX_CONCURRENT	8	Thread pool worker limit
MAX_PAGES	200	Maximum pages crawled
MAX_CRAWL_DEPTH	5	Maximum BFS depth
RESPONSE_BODY_TRUNCATE	8000	Memory protection
DEFAULT_TIMEOUT	10s	Request timeout
Retry backoff	0.5s exponential	Avoids flooding on errors

8.3 Data Handling

• Scan results may contain sensitive data (SQL dumps, file contents). Results are stored locally in results/ with no automatic transmission
• Model files (model.pkl) contain only statistical model weights, no raw data
• PDF reports may contain payload/response excerpts; treat as confidential
• Discord notifications transmit only aggregate counts, never raw payloads

8.4 Legal Compliance

Regulation	Applicability	IntelliScan Compliance
Loi 09-08 (Morocco)	Processing scan data containing PII	Results stored locally; user responsible for data retention policy
RGPD/GDPR (EU)	Scanning EU-hosted targets	No personal data collected beyond HTTP responses; data minimization via truncation
CFAA (US)	Unauthorized computer access	Tool requires explicit --target and --auth; documentation mandates written authorization
Computer Misuse Act (UK)	Unauthorized access	Same as CFAA compliance

---

9. Test Strategy

9.1 Unit Test Coverage Plan

Module	Test File	Key Tests	Coverage Target
Analyzer	test_analyzer.py	SQLi detection, XSS reflection, LFI signatures, severity mapping	>90%
Classifier	test_classifier.py	Feature extraction, training on mock data, model persistence	>85%
Payload Gen	test_payload_gen.py	Each mutation technique, deduplication, type-specific filtering	>90%
Reporter	test_reporter.py	PDF generation, section creation, sanitization	>80%
Crawler	(planned)	BFS traversal, form extraction, blacklist, same-domain	>85%
Injector	(planned)	Task building, concurrent execution, CSRF handling	>80%

9.2 Integration Test Strategy

Docker-based integration testing:

┌────────────┐      HTTP       ┌──────────┐
│ IntelliScan│ ◄──────────────►│   DVWA   │
│  (pytest)  │    port 8080    │ (Docker) │
└────────────┘                 └──────────┘

1. Start DVWA via Docker Compose
2. Run full scan pipeline against http://localhost:8080
3. Verify: crawl discovers ≥4 forms, injector produces results, analyzer labels correctly
4. Verify: ML trains with accuracy ≥90%, report PDF is generated

9.3 Test Fixtures (conftest.py)

Fixture	Content	Used By
sample_injection_results	4 mock injection entries (2 sqli, 1 xss, 1 lfi)	Analyzer, Classifier
sample_labeled_results	4 labeled entries (2 VULNERABLE, 2 NOT_VULNERABLE)	Classifier, Reporter

9.4 CI/CD Pipeline (GitHub Actions)

# .github/workflows/ci.yml
Strategy:
  matrix:
    python-version: [3.10, 3.11, 3.12]

Steps:
  1. Checkout code
  2. Setup Python ${{ matrix.python-version }}
  3. Install dependencies (pip install -e .[dev])
  4. Run linters (ruff check, black --check, mypy)
  5. Run unit tests (pytest --cov=intelliscan --cov-report=xml)
  6. Upload coverage report
  7. Build Docker image (docker build .)

---

10. Deployment Design

10.1 Docker Compose Architecture

┌─────────────────────────────────────────────────────────┐
│              Docker Compose Network                     │
│              (intelliscan-net, bridge)                   │
│                                                         │
│  ┌─────────────────────┐   ┌─────────────────────┐     │
│  │    intelliscan       │   │       dvwa           │     │
│  │                      │   │                      │     │
│  │  Python 3.10+        │   │  Apache + PHP        │     │
│  │  Flask :5000  ◄──────┼───┼──  :80 (ext :8080)   │     │
│  │  CLI (Click)         │   │  MySQL embedded      │     │
│  │                      │   │                      │     │
│  │  Volumes:            │   │  Image:              │     │
│  │   ./results:/app/res │   │  vulnerables/        │     │
│  │   ./models:/app/mod  │   │  web-dvwa:latest     │     │
│  └─────────────────────┘   └─────────────────────┘     │
│         :5000                       :8080                │
└─────────┬───────────────────────────┬───────────────────┘
          │                           │
      Host port 5000             Host port 8080

10.2 Environment Variables

Variable	Default	Description
INTELLISCAN_TIMEOUT	10	HTTP request timeout (seconds)
INTELLISCAN_RESULTS	./results	Results directory path

10.3 Volume Mounts

Host Path	Container Path	Purpose
./results	/app/results	Persist scan results between container restarts
./models	/app/models	Persist trained ML models

10.4 Scaling Considerations

• Horizontal: v3.0 roadmap includes Celery + Redis distributed scanning
• Vertical: MAX_CONCURRENT adjustable from 1 to 32 workers
• Storage: JSON files are stateless; multiple scan runs can coexist with unique filenames

---

11. Technology Choices Justification

11.1 Python 3.10+

Reason	Detail
Type hints	Full PEP 604 union syntax (str | None), PEP 585 generics
Match statement	PEP 634 structural pattern matching (future use)
Ecosystem	scikit-learn, requests, BeautifulSoup — mature, well-documented
Academic context	Most taught language in cybersecurity curricula

11.2 Random Forest (scikit-learn)

Reason	Detail
Interpretability	feature_importances_ enables academic analysis
No scaling required	Tree-based models handle heterogeneous features natively
Ensemble robustness	100 trees reduce variance vs single decision tree
Balanced class weights	Built-in handling of class imbalance
Reproducibility	random_state=42 ensures deterministic results

11.3 fpdf2

Reason	Detail
Pure Python	No system dependencies (unlike ReportLab, WeasyPrint)
Lightweight	~200KB package, no C extensions
Sufficient features	Tables, colored cells, multi-cell, header/footer hooks
License	LGPL — compatible with MIT project license

11.4 Flask

Reason	Detail
Lightweight	Minimal framework, ideal for prototype dashboard
Familiar	Widely used in academic projects
REST-friendly	Easy API endpoint creation for future integrations
Jinja2 templates	Built-in templating for dashboard views

11.5 Click + Rich

Reason	Detail
Click	Decorator-based CLI definition, automatic help generation, type validation
Rich	Colored output, tables, progress bars, tracebacks — professional DX
Combination	RichHandler integrates logging with Rich console output

---

12. Limitations & Future Work

12.1 Current Limitations

#	Limitation	Impact	Mitigation
1	LFI detection: ~10% on DVWA	PHP realpath() in Docker blocks traversal	Higher rates on production configs; expand LFI signatures
2	ML trained on DVWA only	Model may not generalize to other apps	Retrain on diverse datasets; transfer learning in v2.0
3	Hardcoded DVWA_FORMS	Fallback forms specific to DVWA endpoints	Crawler dynamically discovers forms in real scans
4	No blind SQLi	Time-based and boolean-based SQLi not detected	Planned for v1.1
5	No DOM XSS	Client-side JavaScript execution not analyzed	Selenium-based detection in v2.0
6	Single-machine only	No distributed scanning capability	Celery + Redis in v3.0

12.2 Roadmap

v1.1 — Extended Detection (Near-term)

• CSRF detection module
• Command Injection detection
• SSRF detection
• GraphQL endpoint support
• OAuth2/JWT authentication
• Blind SQLi (time-based inference)

v2.0 — Advanced ML (Medium-term)

• LSTM sequence classifier for payload/response analysis
• Active learning loop: analyst labels → model retraining
• DOM XSS detection via Selenium headless browser
• CI/CD plugins: GitHub Action, Jenkins plugin
• Multi-target scanning with priority queue

v3.0 — Enterprise Scale (Long-term)

• Distributed scanning with Celery + Redis task queue
• CVE knowledge base integration (NVD, MITRE)
• Auto-remediation suggestions via LLM (GPT-4 / local models)
• Web-based management console with user roles
• Compliance report templates (PCI-DSS, SOC 2)

---

Appendix A — Generating Architecture Diagrams

All diagrams referenced in this document can be generated as high-resolution PNG files using the provided script.

Prerequisites:

pip install matplotlib graphviz

Additionally install the Graphviz system package:

• Windows: winget install graphviz or download from https://graphviz.org/download/
• Linux: sudo apt install graphviz
• macOS: brew install graphviz

Generate all diagrams:

python docs/generate_diagrams.py

Output files in docs/images/:

File	Diagram
01_pipeline_diagram.png	6-module sequential pipeline
02_dependency_graph.png	Module dependency graph
03_data_flow_diagram.png	Data flow between modules
04_ml_feature_importance.png	RF feature importance bar chart
05_class_diagram.png	Class/component UML diagram
06_deployment_diagram.png	Docker Compose architecture
07_ml_training_flow.png	ML training & evaluation pipeline
08_mutation_pipeline.png	Payload mutation techniques

If Graphviz is not installed, DOT source files (.dot) are saved for manual rendering.

---

Appendix B — References

1. OWASP Top 10 (2021). https://owasp.org/Top10/
2. Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5-32.
3. DVWA — Damn Vulnerable Web Application. https://github.com/digininja/DVWA
4. scikit-learn documentation. https://scikit-learn.org/stable/
5. fpdf2 documentation. https://py-pdf.github.io/fpdf2/
6. Click documentation. https://click.palletsprojects.com/
7. Rich documentation. https://rich.readthedocs.io/

---

Document generated for the Master's thesis (PFE) at Ibn Tofail University, Kenitra, Morocco.
Supervisor: Pr. Youssef FAKHRI — Academic year 2025–2026.