docs/ARCHITECTURE.md

IntelliScan Architecture

Design Philosophy

IntelliScan is built on three core principles:

1. Modularity: each scanning phase is an independent module connected via JSON files. This allows running, testing, and replacing modules independently.
2. Reproducibility: deterministic settings (random_state=42, fixed payload sets, Docker target) ensure that scans can be repeated with identical results.
3. Extensibility: new vulnerability types, payloads, and detection algorithms can be added without modifying existing code.

High-level Architecture

                    +-------------------+
                    |   User / CI / UI  |
                    +---------+---------+
                              |
                  +-----------+----------+
                  |  CLI / Web Dashboard |
                  +-----------+----------+
                              |
                  +-----------+----------+
                  |  IntelliScan Core    |  <-- Pipeline orchestrator
                  +-----------+----------+
                              |
        +----------+----------+---------+----------+----------+
        |          |          |         |          |          |
   +----+----+ +---+---+ +----+----+ +--+--+  +----+----+ +---+----+
   | Crawler | |Inject.| |Analyzer | |  ML | | PayloadG | |Reporter|
   +---------+ +-------+ +---------+ +-----+  +----------+ +--------+
        |          |          |         |          |          |
        v          v          v         v          v          v
  results.json  inj.json  labeled.json model.pkl  payloads.json report.pdf

Module Responsibilities

1. Crawler (intelliscan/modules/crawler.py)

Input: target URL, optional credentials. Output: results.json listing forms, URL parameters, visited pages.

• BFS exploration with configurable depth (MAX_CRAWL_DEPTH)
• Authenticates via the target's login form, extracting CSRF tokens
• Sets DVWA security level to low if applicable
• Strips URL fragments (#) before queuing — fixes a common scanner bug
• Filters out blacklisted paths (logout, setup, .git/...)
• Returns dataclasses (Form, UrlParam) for type safety

2. Injector (intelliscan/modules/injector.py)

Input: target URL, payloads from payloads/*.txt. Output: injection_results.json — request + response data.

• Concurrent injection via ThreadPoolExecutor (default 8 workers)
• Per-request CSRF token retrieval for POST endpoints
• Truncates response bodies to 8000 chars (preserves all needed signatures)
• Falls back to hardcoded DVWA_FORMS definitions if crawler missed Submit buttons
• Records: target_url, method, vuln_type, param, payload, status_code, response_len, response_body

3. Analyzer (intelliscan/modules/analyzer.py)

Input: injection_results.json. Output: labeled_results.json — same data + label (VULNERABLE/NOT_VULNERABLE), reason, severity.

Per-type heuristics:

Type	Detection logic
SQLi	Count "first name:" occurrences (DVWA dump signature); fallback to SQL_ERROR_PATTERNS from config
XSS	Check if payload appears verbatim in body; secondary check on dangerous patterns (<script, onerror=, alert()
LFI	Search for LFI_SIGNATURES (root:x:0:0, daemon:x:, /bin/bash, etc.)

4. ML Classifier (intelliscan/modules/classifier.py)

Input: labeled_results.json. Output: trained model.pkl + TrainingReport with metrics.

Feature extraction (11 features)

F1.  response_len            (numeric)
F2.  status_code             (numeric)
F3.  payload_len             (numeric)
F4.  vuln_type_encoded       (categorical: sqli=0, xss=1, lfi=2)
F5.  has_sql_error           (binary)
F6.  has_first_name          (binary)
F7.  has_script_tag          (binary)
F8.  has_alert               (binary)
F9.  has_passwd              (binary)
F10. has_onerror             (binary)
F11. payload_in_response     (binary)

Random Forest configuration

• n_estimators=100
• class_weight='balanced' to handle imbalanced datasets
• random_state=42 for reproducibility
• Train/test split: 80/20 with stratification
• 5-fold cross-validation for robust performance estimation

Why Random Forest?

• Resistance to overfitting on small datasets (Breiman 2001 convergence theorem)
• Interpretable via feature importances (Gini Mean Decrease in Impurity)
• Strong baseline documented in literature (Alghawazi et al. 2022, Irungu et al. 2023)
• No hyperparameter tuning required — works well out of the box

5. Payload Generator (intelliscan/modules/payload_gen.py)

Input: base payloads from payloads/*.txt. Output: all_payloads.json with mutated variants.

Six mutation techniques:

#	Technique	Example
1	Case alternation	OR -> oR
2	SQL comment insertion	OR 1=1 -> OR/**/1=1
3	Quote substitution	' -> "
4	URL encoding	' -> %27
5	Double URL encoding	' -> %2527
6	Whitespace variation	space -> tab/double-space//**/

Coverage gain: 24 base payloads -> 135 variants (+462%).

6. Reporter (intelliscan/modules/reporter.py)

Input: labeled_results.json + scan metadata. Output: PDF report (5 sections).

• Cover page: target, date, severity boxes
• Executive Summary: stats by vulnerability type
• Detailed Findings: each VULNERABLE entry with URL, payload, evidence
• Recommendations: per-type remediation guidance
• Methodology: pipeline description + ML metrics

Built with fpdf2 for native Python PDF generation (no system dependencies).

Data Flow

The pipeline communicates via JSON files for transparency and debugging:

target URL -> Crawler -> results.json
                          |
                          v
              [DVWA forms hardcoded as fallback]
                          |
                          v
               Injector -> injection_results.json
                          |
                          v
               Analyzer -> labeled_results.json
                          |
                          +---> ML Classifier -> model.pkl
                          |
                          +---> Reporter -> report.pdf

Each module can be re-run independently:

# Re-train ML with different hyperparameters without re-scanning
python -m intelliscan train --dataset labeled_results.json

# Re-generate report with custom branding without re-scanning
python -m intelliscan.modules.reporter --input labeled_results.json

Concurrency Model

• Crawler: sequential (BFS with shared visited set)
• Injector: concurrent (ThreadPoolExecutor, max 8 workers)
• Analyzer: sequential (CPU-bound, fast)
• Classifier: parallel internally (n_jobs=-1 in scikit-learn)
• Reporter: sequential

Network I/O is the bottleneck, hence the concurrency in the Injector.

Configuration

All tunable parameters are centralized in intelliscan/config.py:

DEFAULT_TIMEOUT     = 10        # HTTP timeout in seconds
MAX_CONCURRENT      = 8         # ThreadPoolExecutor workers
MAX_CRAWL_DEPTH     = 5         # BFS depth limit
RF_N_ESTIMATORS     = 100       # Number of trees
RF_RANDOM_STATE     = 42        # Reproducibility seed
TRAIN_TEST_SPLIT    = 0.2       # Test set ratio
CV_FOLDS            = 5         # Cross-validation folds

Override via environment variables:

• INTELLISCAN_TIMEOUT
• INTELLISCAN_RESULTS (output directory)

Extension Points

To add a new vulnerability type (e.g., CSRF):

1. Add payloads in payloads/csrf.txt
2. Update config.py:
   • Add to PAYLOAD_FILES
   • Add CSRF form definition to DVWA_FORMS
   • Add severity to SEVERITY dict
3. Implement _detect_csrf() in analyzer.py and wire into _classify()
4. (Optional) Add CSRF-specific features to MLClassifier.extract_features()
5. Add tests in tests/test_analyzer.py

The modular design means no other modules need changes.

Web Dashboard

The Flask dashboard (intelliscan/web/app.py) provides:

• POST /api/scan: launch a scan asynchronously (returns scan_id)
• GET /api/scan/: poll scan status
• GET /api/report/: download the PDF report
• GET /api/results: latest labeled results

A background thread runs the scan; results are stored in an in-memory registry. For production deployments, replace with Redis + Celery.

Security Considerations

IntelliScan itself follows secure coding practices:

• TLS verification enabled by default (can be disabled with --no-tls-verify)
• Configurable rate limiting (RATE_LIMIT_DELAY)
• HTTP retry logic with exponential backoff
• No payload execution — only HTTP requests are sent
• Non-root Docker user

Ethical use: see LICENSE and the disclaimer in README.md.