docs/ARCHITECTURE.md

# IntelliScan Architecture

## Design Philosophy

IntelliScan is built on three core principles:

1. **Modularity**: each scanning phase is an independent module connected via JSON files. This allows running, testing, and replacing modules independently.
2. **Reproducibility**: deterministic settings (`random_state=42`, fixed payload sets, Docker target) ensure that scans can be repeated with identical results.
3. **Extensibility**: new vulnerability types, payloads, and detection algorithms can be added without modifying existing code.

## High-level Architecture

```
                    +-------------------+
                    |   User / CI / UI  |
                    +---------+---------+
                              |
                  +-----------+----------+
                  |  CLI / Web Dashboard |
                  +-----------+----------+
                              |
                  +-----------+----------+
                  |  IntelliScan Core    |  <-- Pipeline orchestrator
                  +-----------+----------+
                              |
        +----------+----------+---------+----------+----------+
        |          |          |         |          |          |
   +----+----+ +---+---+ +----+----+ +--+--+  +----+----+ +---+----+
   | Crawler | |Inject.| |Analyzer | |  ML | | PayloadG | |Reporter|
   +---------+ +-------+ +---------+ +-----+  +----------+ +--------+
        |          |          |         |          |          |
        v          v          v         v          v          v
  results.json  inj.json  labeled.json model.pkl  payloads.json report.pdf
```

## Module Responsibilities

### 1. Crawler (`intelliscan/modules/crawler.py`)

**Input**: target URL, optional credentials.
**Output**: `results.json` listing forms, URL parameters, visited pages.

- BFS exploration with configurable depth (`MAX_CRAWL_DEPTH`)
- Authenticates via the target's login form, extracting CSRF tokens
- Sets DVWA security level to `low` if applicable
- Strips URL fragments (`#`) before queuing — fixes a common scanner bug
- Filters out blacklisted paths (logout, setup, .git/...)
- Returns dataclasses (`Form`, `UrlParam`) for type safety

### 2. Injector (`intelliscan/modules/injector.py`)

**Input**: target URL, payloads from `payloads/*.txt`.
**Output**: `injection_results.json` — request + response data.

- Concurrent injection via `ThreadPoolExecutor` (default 8 workers)
- Per-request CSRF token retrieval for POST endpoints
- Truncates response bodies to 8000 chars (preserves all needed signatures)
- Falls back to hardcoded `DVWA_FORMS` definitions if crawler missed Submit buttons
- Records: `target_url`, `method`, `vuln_type`, `param`, `payload`, `status_code`, `response_len`, `response_body`

### 3. Analyzer (`intelliscan/modules/analyzer.py`)

**Input**: `injection_results.json`.
**Output**: `labeled_results.json` — same data + `label` (VULNERABLE/NOT_VULNERABLE), `reason`, `severity`.

Per-type heuristics:

| Type | Detection logic |
|------|----------------|
| **SQLi** | Count `"first name:"` occurrences (DVWA dump signature); fallback to `SQL_ERROR_PATTERNS` from config |
| **XSS** | Check if payload appears verbatim in body; secondary check on dangerous patterns (`<script`, `onerror=`, `alert(`) |
| **LFI** | Search for `LFI_SIGNATURES` (`root:x:0:0`, `daemon:x:`, `/bin/bash`, etc.) |

### 4. ML Classifier (`intelliscan/modules/classifier.py`)

**Input**: `labeled_results.json`.
**Output**: trained `model.pkl` + `TrainingReport` with metrics.

#### Feature extraction (11 features)

```
F1.  response_len            (numeric)
F2.  status_code             (numeric)
F3.  payload_len             (numeric)
F4.  vuln_type_encoded       (categorical: sqli=0, xss=1, lfi=2)
F5.  has_sql_error           (binary)
F6.  has_first_name          (binary)
F7.  has_script_tag          (binary)
F8.  has_alert               (binary)
F9.  has_passwd              (binary)
F10. has_onerror             (binary)
F11. payload_in_response     (binary)
```

#### Random Forest configuration

- `n_estimators=100`
- `class_weight='balanced'` to handle imbalanced datasets
- `random_state=42` for reproducibility
- Train/test split: 80/20 with stratification
- 5-fold cross-validation for robust performance estimation

#### Why Random Forest?

- **Resistance to overfitting** on small datasets (Breiman 2001 convergence theorem)
- **Interpretable** via feature importances (Gini Mean Decrease in Impurity)
- **Strong baseline** documented in literature (Alghawazi et al. 2022, Irungu et al. 2023)
- **No hyperparameter tuning required** — works well out of the box

### 5. Payload Generator (`intelliscan/modules/payload_gen.py`)

**Input**: base payloads from `payloads/*.txt`.
**Output**: `all_payloads.json` with mutated variants.

Six mutation techniques:

| # | Technique | Example |
|---|-----------|---------|
| 1 | Case alternation | `OR` -> `oR` |
| 2 | SQL comment insertion | `OR 1=1` -> `OR/**/1=1` |
| 3 | Quote substitution | `'` -> `"` |
| 4 | URL encoding | `'` -> `%27` |
| 5 | Double URL encoding | `'` -> `%2527` |
| 6 | Whitespace variation | space -> tab/double-space/`/**/` |

Coverage gain: **24 base payloads -> 135 variants (+462%)**.

### 6. Reporter (`intelliscan/modules/reporter.py`)

**Input**: `labeled_results.json` + scan metadata.
**Output**: PDF report (5 sections).

- Cover page: target, date, severity boxes
- Executive Summary: stats by vulnerability type
- Detailed Findings: each VULNERABLE entry with URL, payload, evidence
- Recommendations: per-type remediation guidance
- Methodology: pipeline description + ML metrics

Built with **fpdf2** for native Python PDF generation (no system dependencies).

## Data Flow

The pipeline communicates via JSON files for transparency and debugging:

```
target URL -> Crawler -> results.json
                          |
                          v
              [DVWA forms hardcoded as fallback]
                          |
                          v
               Injector -> injection_results.json
                          |
                          v
               Analyzer -> labeled_results.json
                          |
                          +---> ML Classifier -> model.pkl
                          |
                          +---> Reporter -> report.pdf
```

Each module can be re-run independently:

```bash
# Re-train ML with different hyperparameters without re-scanning
python -m intelliscan train --dataset labeled_results.json

# Re-generate report with custom branding without re-scanning
python -m intelliscan.modules.reporter --input labeled_results.json
```

## Concurrency Model

- **Crawler**: sequential (BFS with shared visited set)
- **Injector**: concurrent (`ThreadPoolExecutor`, max 8 workers)
- **Analyzer**: sequential (CPU-bound, fast)
- **Classifier**: parallel internally (`n_jobs=-1` in scikit-learn)
- **Reporter**: sequential

Network I/O is the bottleneck, hence the concurrency in the Injector.

## Configuration

All tunable parameters are centralized in `intelliscan/config.py`:

```python
DEFAULT_TIMEOUT     = 10        # HTTP timeout in seconds
MAX_CONCURRENT      = 8         # ThreadPoolExecutor workers
MAX_CRAWL_DEPTH     = 5         # BFS depth limit
RF_N_ESTIMATORS     = 100       # Number of trees
RF_RANDOM_STATE     = 42        # Reproducibility seed
TRAIN_TEST_SPLIT    = 0.2       # Test set ratio
CV_FOLDS            = 5         # Cross-validation folds
```

Override via environment variables:
- `INTELLISCAN_TIMEOUT`
- `INTELLISCAN_RESULTS` (output directory)

## Extension Points

To add a new vulnerability type (e.g., CSRF):

1. Add payloads in `payloads/csrf.txt`
2. Update `config.py`:
   - Add to `PAYLOAD_FILES`
   - Add CSRF form definition to `DVWA_FORMS`
   - Add severity to `SEVERITY` dict
3. Implement `_detect_csrf()` in `analyzer.py` and wire into `_classify()`
4. (Optional) Add CSRF-specific features to `MLClassifier.extract_features()`
5. Add tests in `tests/test_analyzer.py`

The modular design means no other modules need changes.

## Web Dashboard

The Flask dashboard (`intelliscan/web/app.py`) provides:

- **POST /api/scan**: launch a scan asynchronously (returns scan_id)
- **GET /api/scan/<id>**: poll scan status
- **GET /api/report/<id>**: download the PDF report
- **GET /api/results**: latest labeled results

A background thread runs the scan; results are stored in an in-memory registry. For production deployments, replace with Redis + Celery.

## Security Considerations

IntelliScan itself follows secure coding practices:

- TLS verification enabled by default (can be disabled with `--no-tls-verify`)
- Configurable rate limiting (`RATE_LIMIT_DELAY`)
- HTTP retry logic with exponential backoff
- No payload execution — only HTTP requests are sent
- Non-root Docker user

**Ethical use**: see [LICENSE](../LICENSE) and the disclaimer in [README.md](../README.md).