Hugging Face's logo

Spaces:
simoox
/
Intelliscan
Runtime error

App Files Community
Intelliscan / README.md
simoox's picture
simoox
Upload 204 files
5e0bd20 verified
preview code
|
Raw
History Blame Contribute Delete
15.6 kB
metadata 
title: IntelliScan
emoji: 🛡️
colorFrom: green
colorTo: blue
sdk: docker
pinned: false

IntelliScan

AI-Powered Web Vulnerability Scanner with Machine Learning Classification Automated detection of SQLi, XSS, and LFI vulnerabilities using a hybrid pipeline of signature-based analysis and Random Forest classification.

Python 3.10+ License: MIT Code style: black CI ML OWASP

Overview

IntelliScan is an advanced web application vulnerability scanner that combines traditional signature-based detection with a Machine Learning classifier (Random Forest) to detect and validate security vulnerabilities. It is the result of a final-year Master's project in Cybersecurity & AI at Ibn Tofail University, Kenitra.

Unlike typical scanners that rely solely on static signatures, IntelliScan extracts 11 features from HTTP responses and uses a trained Random Forest model to classify exploitations with 100% F1-Score on the DVWA test environment.

Key Differentiators

  • Hybrid detection: signature-based + ML classification (defense in depth)
  • Dual ML models: a scan-result Random Forest (11 behavioral features, no leakage) plus a CSIC 2010 Random Forest (15 features, trained on 61k real HTTP requests) for independent pre-scan threat scoring
  • Mutation engine: generates WAF-bypassing payload variants using 8 techniques
  • Adaptive crawl-driven injection: targets are resolved from discovered forms and URL parameters, falling back to built-in DVWA definitions
  • Modular pipeline: 6 independent modules connected via JSON files
  • Professional reporting: PDF reports with charts, evidence, and remediation
  • Web dashboard: Flask UI with real per-module progress and recent-scan history
  • Concurrent scanning: ThreadPoolExecutor for 4-8x speedup
  • Docker-ready: full docker-compose stack with target lab

Architecture 

+------------+     +------------+     +-------------+     +-----------+
|  Crawler   | --> |  Injector  | --> |  Analyzer   | --> |  ML Class.|
+------------+     +------------+     +-------------+     +-----------+
      |                  |                   |                  |
      v                  v                   v                  v
  results.json    injection_results    labeled_results      model.pkl
                                              |
                                              v
                                   +------------------+
                                   |  Payload Gen     |
                                   |  (8 mutations)   |
                                   +------------------+
                                              |
                                              v
                                   +------------------+
                                   |  Report Gen      |
                                   |  (PDF + Web)     |
                                   +------------------+

The 6 Modules

# Module Role Output
1 Crawler BFS exploration, CSRF handling, form/URL discovery results.json
2 Injector GET/POST payload injection, response capture injection_results.json
3 Analyzer Signature-based vulnerability detection labeled_results.json
4 ML Classifier Random Forest, 11 behavioral features (+ optional CSIC 15-feature model) model.pkl / csic_model.pkl
5 Payload Generator 8 mutation techniques, adaptive feedback all_payloads.json
6 Report Generator Professional PDF + JSON report report.pdf

Features

Core Detection Capabilities

  • SQL Injection (SQLi): Error-based, Union-based, Boolean Blind, Time-based
  • Cross-Site Scripting (XSS): Reflected, Stored, DOM-based detection
  • Local File Inclusion (LFI): Path traversal, encoding bypass, wrappers
  • Extensible: plugin-based architecture for adding new vulnerability types

Machine Learning Classifiers

Scan-result model (classifier.py) — learns from the live scan:

  • Algorithm: Random Forest (100 trees, balanced class weight, depth-capped)
  • Features: 11 purely behavioral features (response length, status, ratios, URL shape) — deliberately no signature-derived features, so there is no data leakage from the analyzer's labels
  • Training: stratified 80/20 split + 5-fold cross-validation; the per-type response-length statistics are fit on the train split only

CSIC 2010 model (csic_trainer.py) — pre-trained threat scorer:

  • Trained on 61,065 real HTTP requests (36k normal / 25k attack)
  • 15 features: 5 URL, 5 content, 2 request-level, 3 lexical (entropy, longest-param length, digit ratio)
  • Measured performance: Accuracy 90.9%, F1 91.0%, ROC-AUC 0.982, far above the Logistic Regression baseline (73.6%, AUC 0.813)
  • Vectorized feature extraction (no DataFrame.iterrows) for fast retraining
  • Attaches an ml_label / ml_confidence second opinion to every finding

Payload Mutation Engine

Eight mutation techniques generate WAF-bypassing variants:

  1. Case mutation: OR -> oR, Or
  2. SQL comment insertion: OR 1=1 -> OR/**/1=1
  3. Quote substitution: ' -> " for some DBMS
  4. URL encoding: ' -> %27, < -> %3C
  5. Double URL encoding: ' -> %2527
  6. Whitespace variation: spaces -> tabs / /**/
  7. Unicode normalization: SQL keywords -> fullwidth homoglyphs
  8. HTML entity encoding: <script> -> &#60;script&#62;

Adaptive mode records detection feedback and skips techniques with a >80% block rate, focusing effort on mutations the WAF misses.

Reporting

  • Professional PDF reports with cover page, executive summary, vulnerability details, and remediation recommendations
  • Web dashboard (Flask + Chart.js) with real per-module progress, a bento metrics grid, feature-importance bars, and a recent-scans panel
  • JSON exports for CI/CD integration
  • Color-coded terminal output during scan
  • Discord webhook notifications (optional)

Quick Start

Installation 

# Clone the repo
git clone https://github.com/sabkari-mohamed/intelliscan.git
cd intelliscan

# Install dependencies
pip install -r requirements.txt

# Optional: pull DVWA test target
docker run -d -p 8080:80 --name dvwa vulnerables/web-dvwa

Basic Usage 

# Full scan against DVWA
python -m intelliscan scan --target http://localhost:8080 \
    --auth admin:password \
    --report report.pdf

# Discovery only
python -m intelliscan crawl --target http://localhost:8080

# Train the scan-result ML classifier on your own dataset
python -m intelliscan train --dataset labeled_results.json

# Train the CSIC 2010 model (Random Forest + Logistic Regression baseline)
python -m intelliscan train-csic --dataset csic_database.csv

# Generate mutated payloads
python -m intelliscan generate-payloads --output all_payloads.json

Web Dashboard 

python -m intelliscan web --port 5000        # preferred
# or: python -m intelliscan.web.app
# Open http://localhost:5000

Docker Compose (Full Stack) 

docker-compose up
# IntelliScan UI:  http://localhost:5000
# DVWA target:     http://localhost:8080

Performance Benchmarks

Tested on DVWA (security level: low) running in Docker:

Module Metric Value
Crawler Pages discovered 33
Crawler Forms found 16
Injector Total injections 40
Analyzer SQLi detection 10/10 (100%)
Analyzer XSS detection 20/20 (100%)
Analyzer LFI detection 1/10 (10%)*
ML Classifier Accuracy 100%
ML Classifier F1-Score 100%
ML Classifier CV (5-fold) 1.00 ± 0.00
Payload Gen Variants generated +111

*LFI rate is limited by PHP's realpath() protection in Docker; on production-style configurations, the rate is significantly higher.

Comparison with Other Scanners

Feature IntelliScan VulnScan SQLMap OWASP ZAP
SQLi Yes (100%) Yes (basic) Yes (gold std) Yes (~85%)
XSS Yes (100%) Yes (basic) No Yes (~80%)
LFI Yes Yes No Partial
Machine Learning Yes (RF) No No No
Payload mutation Yes (8 tech) No Partial No
PDF reports Yes No No Yes
Web dashboard Yes (Flask) No No Yes
Modular pipeline Yes (6 mods) No No No
Concurrent scan Yes No Yes Yes
Docker support Yes No No Yes

Project Structure 

intelliscan/
├── intelliscan/
│   ├── __init__.py
│   ├── __main__.py             # CLI entry point
│   ├── core.py                 # Pipeline orchestrator
│   ├── config.py               # Settings and constants
│   ├── modules/
│   │   ├── crawler.py          # Module 1: BFS + CSRF
│   │   ├── injector.py         # Module 2: payload injection
│   │   ├── analyzer.py         # Module 3: signature detection
│   │   ├── classifier.py       # Module 4: Random Forest ML
│   │   ├── payload_gen.py      # Module 5: 8 mutation techniques
│   │   └── reporter.py         # Module 6: PDF generation
│   ├── web/
│   │   ├── app.py              # Flask web dashboard
│   │   ├── templates/          # Jinja2 templates
│   │   └── static/             # CSS, JS, images
│   └── utils/
│       ├── http_client.py      # Session, retries, rate limiting
│       └── notifier.py         # Discord webhook
├── tests/
│   ├── test_crawler.py
│   ├── test_injector.py
│   ├── test_analyzer.py
│   ├── test_classifier.py
│   └── test_payload_gen.py
├── payloads/
│   ├── sqli.txt
│   ├── xss.txt
│   └── lfi.txt
├── docs/
│   ├── ARCHITECTURE.md
│   └── images/
├── examples/
│   ├── dvwa_scan.py
│   └── custom_target.py
├── requirements.txt
├── setup.py
├── Dockerfile
├── docker-compose.yml
├── README.md
└── LICENSE

How It Works

1. Crawling Phase

The Crawler authenticates against the target (handling CSRF tokens), then performs a BFS traversal of all accessible pages, extracting forms (action, method, inputs) and URL parameters.

from intelliscan.modules.crawler import Crawler

crawler = Crawler("http://localhost:8080", auth=("admin", "password"))
results = crawler.run()
# {"forms": [...], "url_params": [...]}

2. Injection Phase

The Injector resolves targets from the crawl result (discovered forms and URL parameters), falling back to the built-in DVWA definitions when the crawl finds nothing injectable. Each request's response is captured (status code, length, body excerpt).

from intelliscan.modules.injector import Injector

injector = Injector(crawler.http, target, crawl_result=results)
injections = injector.run_all()

3. Detection Phase

The Analyzer uses type-specific signatures:

  • SQLi: counts "First name:" occurrences (DVWA), checks SQL error patterns
  • XSS: verifies payload reflection in response, checks <script, onerror=, alert(
  • LFI: searches for root:x:0:0, daemon:x:, /bin/bash

4. ML Classification

Eleven behavioral features are extracted from each labeled response. A Random Forest classifier learns to discriminate VULNERABLE vs NOT_VULNERABLE from response/payload shape (length, status, ratios, URL depth) rather than from the analyzer's own signatures — avoiding label leakage. When a pre-trained CSIC model is present, each finding additionally receives an independent ml_label / ml_confidence from the 15-feature CSIC Random Forest.

5. Payload Mutation

The generator applies 8 mutation techniques to base payloads:

from intelliscan.modules.payload_gen import PayloadGenerator

gen = PayloadGenerator()
variants = gen.generate("' OR 1=1--", vuln_type="sqli")
# ["' OR 1=1--", "' Or 1=1--", "' OR/**/1=1--", "%27%20OR%201%3D1--", ...]

6. Reporting

The Reporter produces a 5-section PDF:

  1. Cover page with executive summary
  2. Statistics by vulnerability type
  3. Detailed findings (URL, payload, evidence)
  4. Remediation recommendations
  5. Methodology and disclaimer

Ethical Use & Disclaimer

IntelliScan is intended exclusively for authorized security testing. Unauthorized scanning of systems you do not own is illegal in most jurisdictions (Loi 09-08 in Morocco, RGPD in EU, CFAA in USA).

By using this tool, you confirm that:

  1. You have explicit written authorization to test the target system
  2. You will not use IntelliScan for malicious purposes
  3. You accept full responsibility for your usage

The authors disclaim any liability for misuse. Always test against legal targets like DVWA, WebGoat, OWASP Juice Shop, or HackTheBox machines.

Roadmap

Short term (v1.x)

  • Add CSRF, Command Injection, SSRF detection
  • Support for GraphQL endpoints
  • OAuth2/JWT authentication
  • Improved Blind SQLi detection (time-based)

Medium term (v2.x)

  • LSTM model for sequence-aware classification
  • Active learning loop (uncertain predictions -> human review)
  • Browser-based DOM XSS testing (Selenium)
  • CI/CD plugin (GitHub Action, Jenkins)

Long term (v3.x)

  • Distributed scanning (Celery + Redis)
  • Knowledge base of CVEs and exploits
  • Auto-remediation suggestions via LLM

References

This project builds on the following academic work:

  • Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5-32.
  • Alghawazi, M. et al. (2022). Detection of SQL injection using ML techniques. Journal of Cybersecurity and Privacy.
  • Tadhani, T. et al. (2024). Securing web applications using hybrid deep learning. Scientific Reports (Nature).
  • OWASP Foundation. (2021). OWASP Top 10. https://owasp.org/www-project-top-ten/
  • Pedregosa, F. et al. (2011). Scikit-learn: Machine Learning in Python. JMLR, 12, 2825-2830.

Full bibliography in docs/REFERENCES.md.

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for guidelines.

# Setup dev environment
pip install -r requirements-dev.txt
pre-commit install

# Run tests
pytest tests/ --cov=intelliscan

# Run linter
ruff check intelliscan/
black intelliscan/

License

MIT License. See LICENSE.

Author

SABKARI Mohamed Master in Cybersecurity & Artificial Intelligence Ibn Tofail University, Kenitra, Morocco 2025-2026

Built as a Master's thesis project (PFE) under the supervision of Pr. Youssef FAKHRI.

Acknowledgments

  • The OWASP community for the Top 10 framework and DVWA
  • The Scikit-learn team (Pedregosa et al., 2011) for the ML library
  • The Python and open-source community
  • Pr. Youssef FAKHRI for academic supervision and guidance