docs/urlscan_api.md

URLScan.io API Integration

This module provides a Python client for interacting with the URLScan.io API to analyze URLs for phishing detection.

Setup

1. Get an API key:

   • Sign up at https://urlscan.io/user/signup
   • Get your API key from your account settings

2. Set environment variable:

   export URLSCAN_API_KEY="your_api_key_here"
   

   Or create a .env file:

   cp .env.example .env
   # Edit .env and add your API key
   

Basic Usage

Initialize Client

from phising_detection.api import URLScanClient

# Using environment variable
client = URLScanClient()

# Or pass API key directly
client = URLScanClient(api_key="your_api_key")

Submit a URL for Scanning

# Submit URL
result = client.submit_url(
    url="https://suspicious-site.com",
    visibility="public",  # or "unlisted" or "private"
    tags=["phishing", "test"]
)

uuid = result["uuid"]
print(f"Scan UUID: {uuid}")

Retrieve Results

# Get results by UUID
scan_result = client.get_result(uuid)

# Access scan data
page_title = scan_result["page"]["title"]
screenshot = scan_result["task"]["screenshotURL"]

Submit and Wait for Results

# Submit and automatically wait for completion
result = client.submit_and_wait(
    url="https://example.com",
    max_wait=60,  # seconds
    poll_interval=5  # seconds between checks
)

Get Verdict

# Get simple verdict (malicious/safe)
verdict = client.get_verdict(uuid)
print(f"Verdict: {verdict}")  # "malicious" or "safe"

Search Existing Scans

# Search for scans by domain
results = client.search(
    query="domain:example.com",
    size=10
)

for scan in results["results"]:
    print(scan["task"]["url"])

API Response Examples

Submission Response

{
  "uuid": "abc123...",
  "result": "https://urlscan.io/result/abc123.../",
  "api": "https://urlscan.io/api/v1/result/abc123.../"
}

Result Response

{
  "page": {
    "url": "https://example.com",
    "title": "Example Domain",
    "status": "200"
  },
  "verdicts": {
    "overall": {
      "score": 0,
      "malicious": false
    }
  },
  "task": {
    "uuid": "abc123...",
    "time": "2024-01-01T12:00:00.000Z",
    "screenshotURL": "https://..."
  }
}

Error Handling

from phising_detection.api import URLScanClient, URLScanError

try:
    client = URLScanClient()
    result = client.submit_url("https://example.com")
except URLScanError as e:
    print(f"Error: {e}")

Rate Limits

• Free tier: 50 submissions per day
• Paid tier: Higher limits available
• The client handles rate limit errors automatically

Integration with Phishing Detection

from phising_detection.api import URLScanClient
from phising_detection.data import load_phishing_urls

# Load your phishing URLs
df = load_phishing_urls()

# Analyze URLs
client = URLScanClient()

for idx, row in df.head(10).iterrows():  # Sample first 10
    try:
        result = client.submit_and_wait(row['url'])
        verdict = client.get_verdict(result['task']['uuid'])
        print(f"{row['url']}: {verdict}")
    except URLScanError as e:
        print(f"Error scanning {row['url']}: {e}")

API Documentation

Full API documentation: https://urlscan.io/docs/api/