README.md

---
title: IntelliScan
emoji: 🛡️
colorFrom: green
colorTo: blue
sdk: docker
pinned: false
---

# IntelliScan

> **AI-Powered Web Vulnerability Scanner with Machine Learning Classification**
> Automated detection of SQLi, XSS, and LFI vulnerabilities using a hybrid pipeline of signature-based analysis and Random Forest classification.

[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![CI](https://img.shields.io/badge/CI-passing-brightgreen.svg)]()
[![ML](https://img.shields.io/badge/ML-Random%20Forest-orange.svg)]()
[![OWASP](https://img.shields.io/badge/OWASP-Top%2010-red.svg)]()

---

## Overview

**IntelliScan** is an advanced web application vulnerability scanner that combines traditional signature-based detection with a Machine Learning classifier (Random Forest) to detect and validate security vulnerabilities. It is the result of a final-year Master's project in Cybersecurity & AI at Ibn Tofail University, Kenitra.

Unlike typical scanners that rely solely on static signatures, IntelliScan extracts 11 features from HTTP responses and uses a trained Random Forest model to classify exploitations with 100% F1-Score on the DVWA test environment.

### Key Differentiators

- **Hybrid detection**: signature-based + ML classification (defense in depth)
- **Dual ML models**: a scan-result Random Forest (11 behavioral features, no
  leakage) plus a CSIC 2010 Random Forest (15 features, trained on 61k real
  HTTP requests) for independent pre-scan threat scoring
- **Mutation engine**: generates WAF-bypassing payload variants using 8 techniques
- **Adaptive crawl-driven injection**: targets are resolved from discovered forms
  and URL parameters, falling back to built-in DVWA definitions
- **Modular pipeline**: 6 independent modules connected via JSON files
- **Professional reporting**: PDF reports with charts, evidence, and remediation
- **Web dashboard**: Flask UI with real per-module progress and recent-scan history
- **Concurrent scanning**: ThreadPoolExecutor for 4-8x speedup
- **Docker-ready**: full docker-compose stack with target lab

---

## Architecture

```
+------------+     +------------+     +-------------+     +-----------+
|  Crawler   | --> |  Injector  | --> |  Analyzer   | --> |  ML Class.|
+------------+     +------------+     +-------------+     +-----------+
      |                  |                   |                  |
      v                  v                   v                  v
  results.json    injection_results    labeled_results      model.pkl
                                              |
                                              v
                                   +------------------+
                                   |  Payload Gen     |
                                   |  (8 mutations)   |
                                   +------------------+
                                              |
                                              v
                                   +------------------+
                                   |  Report Gen      |
                                   |  (PDF + Web)     |
                                   +------------------+
```

### The 6 Modules

| # | Module | Role | Output |
|---|--------|------|--------|
| 1 | **Crawler** | BFS exploration, CSRF handling, form/URL discovery | `results.json` |
| 2 | **Injector** | GET/POST payload injection, response capture | `injection_results.json` |
| 3 | **Analyzer** | Signature-based vulnerability detection | `labeled_results.json` |
| 4 | **ML Classifier** | Random Forest, 11 behavioral features (+ optional CSIC 15-feature model) | `model.pkl` / `csic_model.pkl` |
| 5 | **Payload Generator** | 8 mutation techniques, adaptive feedback | `all_payloads.json` |
| 6 | **Report Generator** | Professional PDF + JSON report | `report.pdf` |

---

## Features

### Core Detection Capabilities

- **SQL Injection (SQLi)**: Error-based, Union-based, Boolean Blind, Time-based
- **Cross-Site Scripting (XSS)**: Reflected, Stored, DOM-based detection
- **Local File Inclusion (LFI)**: Path traversal, encoding bypass, wrappers
- **Extensible**: plugin-based architecture for adding new vulnerability types

### Machine Learning Classifiers

**Scan-result model** (`classifier.py`) — learns from the live scan:

- Algorithm: **Random Forest** (100 trees, balanced class weight, depth-capped)
- Features: 11 purely *behavioral* features (response length, status, ratios,
  URL shape) — deliberately no signature-derived features, so there is no data
  leakage from the analyzer's labels
- Training: stratified 80/20 split + 5-fold cross-validation; the per-type
  response-length statistics are fit on the **train split only**

**CSIC 2010 model** (`csic_trainer.py`) — pre-trained threat scorer:

- Trained on **61,065 real HTTP requests** (36k normal / 25k attack)
- 15 features: 5 URL, 5 content, 2 request-level, 3 lexical (entropy,
  longest-param length, digit ratio)
- Measured performance: **Accuracy 90.9%, F1 91.0%, ROC-AUC 0.982**, far above
  the Logistic Regression baseline (73.6%, AUC 0.813)
- Vectorized feature extraction (no `DataFrame.iterrows`) for fast retraining
- Attaches an `ml_label` / `ml_confidence` second opinion to every finding

### Payload Mutation Engine

Eight mutation techniques generate WAF-bypassing variants:

1. **Case mutation**: `OR` -> `oR`, `Or`
2. **SQL comment insertion**: `OR 1=1` -> `OR/**/1=1`
3. **Quote substitution**: `'` -> `"` for some DBMS
4. **URL encoding**: `'` -> `%27`, `<` -> `%3C`
5. **Double URL encoding**: `'` -> `%2527`
6. **Whitespace variation**: spaces -> tabs / `/**/`
7. **Unicode normalization**: SQL keywords -> fullwidth homoglyphs
8. **HTML entity encoding**: `<script>` -> `&#60;script&#62;`

Adaptive mode records detection feedback and skips techniques with a >80% block
rate, focusing effort on mutations the WAF misses.

### Reporting

- Professional **PDF reports** with cover page, executive summary, vulnerability details, and remediation recommendations
- **Web dashboard** (Flask + Chart.js) with real per-module progress, a bento
  metrics grid, feature-importance bars, and a recent-scans panel
- **JSON exports** for CI/CD integration
- **Color-coded terminal** output during scan
- **Discord webhook** notifications (optional)

---

## Quick Start

### Installation

```bash
# Clone the repo
git clone https://github.com/sabkari-mohamed/intelliscan.git
cd intelliscan

# Install dependencies
pip install -r requirements.txt

# Optional: pull DVWA test target
docker run -d -p 8080:80 --name dvwa vulnerables/web-dvwa
```

### Basic Usage

```bash
# Full scan against DVWA
python -m intelliscan scan --target http://localhost:8080 \
    --auth admin:password \
    --report report.pdf

# Discovery only
python -m intelliscan crawl --target http://localhost:8080

# Train the scan-result ML classifier on your own dataset
python -m intelliscan train --dataset labeled_results.json

# Train the CSIC 2010 model (Random Forest + Logistic Regression baseline)
python -m intelliscan train-csic --dataset csic_database.csv

# Generate mutated payloads
python -m intelliscan generate-payloads --output all_payloads.json
```

### Web Dashboard

```bash
python -m intelliscan web --port 5000        # preferred
# or: python -m intelliscan.web.app
# Open http://localhost:5000
```

### Docker Compose (Full Stack)

```bash
docker-compose up
# IntelliScan UI:  http://localhost:5000
# DVWA target:     http://localhost:8080
```

---

## Performance Benchmarks

Tested on DVWA (security level: low) running in Docker:

| Module | Metric | Value |
|---|---|---|
| Crawler | Pages discovered | 33 |
| Crawler | Forms found | 16 |
| Injector | Total injections | 40 |
| Analyzer | SQLi detection | 10/10 (100%) |
| Analyzer | XSS detection | 20/20 (100%) |
| Analyzer | LFI detection | 1/10 (10%)* |
| ML Classifier | Accuracy | 100% |
| ML Classifier | F1-Score | 100% |
| ML Classifier | CV (5-fold) | 1.00 ± 0.00 |
| Payload Gen | Variants generated | +111 |

*LFI rate is limited by PHP's `realpath()` protection in Docker; on production-style configurations, the rate is significantly higher.

---

## Comparison with Other Scanners

| Feature | IntelliScan | VulnScan | SQLMap | OWASP ZAP |
|---|:---:|:---:|:---:|:---:|
| SQLi | Yes (100%) | Yes (basic) | Yes (gold std) | Yes (~85%) |
| XSS | Yes (100%) | Yes (basic) | No | Yes (~80%) |
| LFI | Yes | Yes | No | Partial |
| **Machine Learning** | **Yes (RF)** | No | No | No |
| **Payload mutation** | **Yes (8 tech)** | No | Partial | No |
| **PDF reports** | **Yes** | No | No | Yes |
| **Web dashboard** | **Yes (Flask)** | No | No | Yes |
| **Modular pipeline** | **Yes (6 mods)** | No | No | No |
| **Concurrent scan** | Yes | No | Yes | Yes |
| **Docker support** | Yes | No | No | Yes |

---

## Project Structure

```
intelliscan/
├── intelliscan/
│   ├── __init__.py
│   ├── __main__.py             # CLI entry point
│   ├── core.py                 # Pipeline orchestrator
│   ├── config.py               # Settings and constants
│   ├── modules/
│   │   ├── crawler.py          # Module 1: BFS + CSRF
│   │   ├── injector.py         # Module 2: payload injection
│   │   ├── analyzer.py         # Module 3: signature detection
│   │   ├── classifier.py       # Module 4: Random Forest ML
│   │   ├── payload_gen.py      # Module 5: 8 mutation techniques
│   │   └── reporter.py         # Module 6: PDF generation
│   ├── web/
│   │   ├── app.py              # Flask web dashboard
│   │   ├── templates/          # Jinja2 templates
│   │   └── static/             # CSS, JS, images
│   └── utils/
│       ├── http_client.py      # Session, retries, rate limiting
│       └── notifier.py         # Discord webhook
├── tests/
│   ├── test_crawler.py
│   ├── test_injector.py
│   ├── test_analyzer.py
│   ├── test_classifier.py
│   └── test_payload_gen.py
├── payloads/
│   ├── sqli.txt
│   ├── xss.txt
│   └── lfi.txt
├── docs/
│   ├── ARCHITECTURE.md
│   └── images/
├── examples/
│   ├── dvwa_scan.py
│   └── custom_target.py
├── requirements.txt
├── setup.py
├── Dockerfile
├── docker-compose.yml
├── README.md
└── LICENSE
```

---

## How It Works

### 1. Crawling Phase

The Crawler authenticates against the target (handling CSRF tokens), then performs a BFS traversal of all accessible pages, extracting forms (action, method, inputs) and URL parameters.

```python
from intelliscan.modules.crawler import Crawler

crawler = Crawler("http://localhost:8080", auth=("admin", "password"))
results = crawler.run()
# {"forms": [...], "url_params": [...]}
```

### 2. Injection Phase

The Injector resolves targets from the crawl result (discovered forms and URL
parameters), falling back to the built-in DVWA definitions when the crawl finds
nothing injectable. Each request's response is captured (status code, length,
body excerpt).

```python
from intelliscan.modules.injector import Injector

injector = Injector(crawler.http, target, crawl_result=results)
injections = injector.run_all()
```

### 3. Detection Phase

The Analyzer uses type-specific signatures:

- **SQLi**: counts "First name:" occurrences (DVWA), checks SQL error patterns
- **XSS**: verifies payload reflection in response, checks `<script`, `onerror=`, `alert(`
- **LFI**: searches for `root:x:0:0`, `daemon:x:`, `/bin/bash`

### 4. ML Classification

Eleven behavioral features are extracted from each labeled response. A Random
Forest classifier learns to discriminate VULNERABLE vs NOT_VULNERABLE from
response/payload shape (length, status, ratios, URL depth) rather than from the
analyzer's own signatures — avoiding label leakage. When a pre-trained CSIC
model is present, each finding additionally receives an independent
`ml_label` / `ml_confidence` from the 15-feature CSIC Random Forest.

### 5. Payload Mutation

The generator applies 8 mutation techniques to base payloads:

```python
from intelliscan.modules.payload_gen import PayloadGenerator

gen = PayloadGenerator()
variants = gen.generate("' OR 1=1--", vuln_type="sqli")
# ["' OR 1=1--", "' Or 1=1--", "' OR/**/1=1--", "%27%20OR%201%3D1--", ...]
```

### 6. Reporting

The Reporter produces a 5-section PDF:

1. Cover page with executive summary
2. Statistics by vulnerability type
3. Detailed findings (URL, payload, evidence)
4. Remediation recommendations
5. Methodology and disclaimer

---

## Ethical Use & Disclaimer

**IntelliScan is intended exclusively for authorized security testing.** Unauthorized scanning of systems you do not own is illegal in most jurisdictions (Loi 09-08 in Morocco, RGPD in EU, CFAA in USA).

By using this tool, you confirm that:

1. You have **explicit written authorization** to test the target system
2. You will not use IntelliScan for malicious purposes
3. You accept full responsibility for your usage

The authors disclaim any liability for misuse. **Always test against legal targets** like DVWA, WebGoat, OWASP Juice Shop, or HackTheBox machines.

---

## Roadmap

### Short term (v1.x)

- [ ] Add CSRF, Command Injection, SSRF detection
- [ ] Support for GraphQL endpoints
- [ ] OAuth2/JWT authentication
- [ ] Improved Blind SQLi detection (time-based)

### Medium term (v2.x)

- [ ] LSTM model for sequence-aware classification
- [ ] Active learning loop (uncertain predictions -> human review)
- [ ] Browser-based DOM XSS testing (Selenium)
- [ ] CI/CD plugin (GitHub Action, Jenkins)

### Long term (v3.x)

- [ ] Distributed scanning (Celery + Redis)
- [ ] Knowledge base of CVEs and exploits
- [ ] Auto-remediation suggestions via LLM

---

## References

This project builds on the following academic work:

- Breiman, L. (2001). *Random forests*. Machine Learning, 45(1), 5-32.
- Alghawazi, M. et al. (2022). *Detection of SQL injection using ML techniques*. Journal of Cybersecurity and Privacy.
- Tadhani, T. et al. (2024). *Securing web applications using hybrid deep learning*. Scientific Reports (Nature).
- OWASP Foundation. (2021). *OWASP Top 10*. https://owasp.org/www-project-top-ten/
- Pedregosa, F. et al. (2011). *Scikit-learn: Machine Learning in Python*. JMLR, 12, 2825-2830.

Full bibliography in [docs/REFERENCES.md](docs/REFERENCES.md).

---

## Contributing

Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

```bash
# Setup dev environment
pip install -r requirements-dev.txt
pre-commit install

# Run tests
pytest tests/ --cov=intelliscan

# Run linter
ruff check intelliscan/
black intelliscan/
```

---

## License

MIT License. See [LICENSE](LICENSE).

---

## Author

**SABKARI Mohamed**
Master in Cybersecurity & Artificial Intelligence
Ibn Tofail University, Kenitra, Morocco
2025-2026

Built as a Master's thesis project (PFE) under the supervision of **Pr. Youssef FAKHRI**.

---

## Acknowledgments

- The OWASP community for the Top 10 framework and DVWA
- The Scikit-learn team (Pedregosa et al., 2011) for the ML library
- The Python and open-source community
- Pr. Youssef FAKHRI for academic supervision and guidance