add robots explorer

add defaultdict

fix sitemap

sitemap

max pages to 1000

filename with url paths

remove emojis in app

ssr_mode=False in app

another gradio fix

gradio fix

remove mcp section, improve readability

✨ Upgrade: Improved web scraper with better UI, cleaner content extraction, and human-readable outputs

initial commit