Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | |
| <meta name="description" content="SLM Web Scraper: Local CPU-optimized HTML data scraper compiling targets into JSON layouts offline using BeautifulSoup and Phi-3.5."> | |
| <link rel="canonical" href="https://www.slmagents.ai/web_scraper.html"> | |
| <title>SLM Web Scraper | Documentation</title> | |
| <link rel="stylesheet" href="style.css"> | |
| </head> | |
| <body> | |
| <header> | |
| <div class="container nav-container"> | |
| <div style="display: flex; align-items: center; gap: 10px;"> | |
| <button class="sidebar-toggle" onclick="toggleSidebar()"> | |
| <svg width="24" height="24" fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24"> | |
| <path stroke-linecap="round" stroke-linejoin="round" d="M4 6h16M4 12h16M4 18h16"/> | |
| </svg> | |
| </button> | |
| <a href="index.html" class="logo"> | |
| <div class="logo-icon"> | |
| <svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"> | |
| <path d="M12 2L2 7l10 5 10-5-10-5zM2 17l10 5 10-5M2 12l10 5 10-5"/> | |
| </svg> | |
| </div> | |
| <span>SLM Agents</span> | |
| </a> | |
| </div> | |
| <nav> | |
| <ul> | |
| <li><a href="index.html">Home</a></li> | |
| <li class="dropdown"> | |
| <a class="dropdown-trigger" style="color:var(--primary)"> | |
| Frameworks | |
| <svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"><path d="M6 9l6 6 6-6"/></svg> | |
| </a> | |
| <div class="dropdown-content"> | |
| <a href="orchestrator.html">Orchestrator Docs</a> | |
| <a href="rag.html">RAG Docs</a> | |
| <a href="summarizer.html">Summarizer Docs</a> | |
| <a href="sql.html">Text-to-SQL Docs</a> | |
| <a href="cli.html">CLI Docs</a> | |
| <a href="code_interpreter.html">Code Interpreter Docs</a> | |
| <a href="git_repo_manager.html">Git Repo Manager Docs</a> | |
| <a href="json_cleaner.html">JSON Cleaner Docs</a> | |
| <a href="document_parser.html">Document Parser Docs</a> | |
| <a href="vision_parser.html">Vision Parser Docs</a> | |
| <a href="web_agent.html">Web Agent Docs</a> | |
| <a href="web_scraper.html">Web Scraper Docs</a> | |
| <a href="search_orchestrator.html">Search Orchestrator Docs</a> | |
| </div> | |
| </li> | |
| <li><a href="playground.html">Playground</a></li> | |
| <li><a href="index.html#upcoming">Roadmap</a></li> | |
| <li><a href="https://huggingface.co/spaces/spcv/slm-agents" class="btn-hf-nav" target="_blank" style="display: flex; align-items: center; gap: 6px; padding: 7px 14px; border-radius: 8px; background: #fef3c7; border: 1px solid #f59e0b; color: #b45309; font-weight: 700; text-decoration: none; font-size: 0.85rem; transition: all 0.2s;">🤗 Hugging Face Space</a></li> | |
| </ul> | |
| </nav> | |
| </div> | |
| </header> | |
| <div class="layout-container"> | |
| <aside class="sidebar" id="sidebar"> | |
| <div class="sidebar-group"> | |
| <div class="sidebar-group-title">Active Libraries</div> | |
| <ul class="sidebar-list" id="sidebar-active-list"></ul> | |
| </div> | |
| </aside> | |
| <main class="main-content"> | |
| <div class="doc-content-wrapper"> | |
| <section class="doc-section" style="margin-top: 0;"> | |
| <div class="breadcrumb" style="font-size: 0.85rem; color: #475569; margin-bottom: 1rem; font-weight: 600;"><a href="index.html" style="color: #4f46e5; text-decoration: none;">Home</a> <span style="margin: 0 6px;">›</span> <span style="color: #0f172a;">SLM Web Scraper</span></div> | |
| <div class="badge-pill">🔍 Layout Sanitization</div> | |
| <h1 style="font-size: 2.25rem; color: #0f172a; font-weight: 800; margin-bottom: 0.75rem;">SLM Web Scraper</h1> | |
| <p style="font-size: 1.05rem; color: #334155; line-height: 1.65; font-weight: 500; margin: 0;">Extract structured JSON directly from dirty, obfuscated HTML layouts offline using BeautifulSoup and Phi-3.5-mini.</p> | |
| </section> | |
| <section class="doc-section"> | |
| <h2>🚀 Overview & Capabilities</h2> | |
| <p>The SLM Web Scraper parses raw HTML strings, stripping scripts, styles, duplicate nav tags, and ads. The sanitized flat content is compiled locally into structured formats using target JSON schemas.</p> | |
| </section> | |
| <section class="doc-section" id="install"> | |
| <h2>💻 Installation</h2> | |
| <p>Install the local CPU-optimized package using pip:</p> | |
| <div class="code-panel" style="max-width:100%"> | |
| <div class="code-header"><div class="code-dots"><div class="code-dot"></div><div class="code-dot"></div><div class="code-dot"></div></div><div class="code-title">Terminal</div></div> | |
| <div class="code-content" style="display:block;padding:1.5rem"> | |
| <pre><code><span class="comment"># Install from PyPI</span> | |
| pip install slm-web-scraper</code></pre> | |
| </div> | |
| </div> | |
| </section> | |
| <section class="doc-section"> | |
| <h2>🤖 Truly Agentic Workflow</h2> | |
| <p>The Web Scraper filters raw DOM elements to produce clean extractions:</p> | |
| <ul> | |
| <li><strong>DOM Pruning:</strong> Automatically strips scripts, styles, metadata tags, and ad headers using BeautifulSoup.</li> | |
| <li><strong>Visual Image OCR:</strong> Identifies image tags (<code><img></code>) on the page, downloads them, and runs them through the local <strong>SLMVisionParser</strong> to generate natural language text descriptions, replacing the tags with the descriptions.</li> | |
| <li><strong>Whitespaces Compacting:</strong> Collapses contiguous whitespace blocks into single space delimiters.</li> | |
| <li><strong>Schema Alignment:</strong> Instructs the local Phi-3.5 model to match and structure the sanitized text into clean JSON.</li> | |
| </ul> | |
| </section> | |
| <section class="doc-section"> | |
| <h2>⚡ CPU Performance Tuning Guidelines</h2> | |
| <p>Follow these configuration rules to maximize scraping speed on standard hardware:</p> | |
| <ul> | |
| <li><strong>Sanitize First:</strong> Always clean webpage source code before parsing. Running raw HTML through the model increases token count and latency.</li> | |
| <li><strong>Memory Management:</strong> Clear page memory scopes after heavy parsing cycles to keep footprint under 1.5 GB RAM.</li> | |
| </ul> | |
| </section> | |
| <section class="doc-section"> | |
| <h2>🎯 Accuracy Improvement Tips</h2> | |
| <div class="tip-box"> | |
| <strong>Tip for Layout Cleaning:</strong> Condensing tables or pricing matrices into columnar string lists (e.g. <code>"Item 1 | Price 1"</code>) preserves formatting structure for the model, improving data extraction accuracy. | |
| </div> | |
| <div class="warn-box"> | |
| <strong>Keep Schema Flat:</strong> Keep the target schema flat. Deep nesting can confuse smaller models, causing schema validation errors. | |
| </div> | |
| </section> | |
| <section class="doc-section"> | |
| <h2>API Reference</h2> | |
| <h3>`SLMWebScraper` Initialization</h3> | |
| <pre><code class="language-python">from slm_web_scraper.web_scraper import SLMWebScraper | |
| scraper = SLMWebScraper()</code></pre> | |
| <table class="param-table"> | |
| <thead> | |
| <tr><th>Parameter</th><th>Type</th><th>Description</th></tr> | |
| </thead> | |
| <tbody> | |
| <tr><td>model_path</td><td>str</td><td>Local path to Phi-3.5 weights. Defaults to "../../models/phi-3.5-mini-instruct-onnx".</td></tr> | |
| <tr><td>system_prompt</td><td>str | None</td><td>Optional custom system prompt instructions overriding the default template.</td></tr> | |
| <tr><td>user_input</td><td>str | None</td><td>Optional additional user-supplied target parameters or variables.</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>`scrape_url` Method</h3> | |
| <p>Fetches a URL, filters out header/footer menus, dropdown lists, sidebars, and ads, and automatically parses tables and image content:</p> | |
| <h4>Hybrid Visual Scraping (Tables & Images)</h4> | |
| <p>This example scrapes the vision parser documentation page <code>https://www.slmagents.ai/vision_parser.html</code>, describing its parameters table and flowchart image automatically:</p> | |
| <pre><code class="language-python">from slm_web_scraper.web_scraper import SLMWebScraper | |
| scraper = SLMWebScraper() | |
| # Scrapes page and converts tables/images to descriptions automatically | |
| clean_text = scraper.scrape_url("https://www.slmagents.ai/vision_parser.html") | |
| print(clean_text)</code></pre> | |
| <div class="tip-box"> | |
| <strong>Cleaned Webpage Text Output with Visual Descriptions:</strong><br> | |
| <pre><code>SLM Vision Parser | Documentation | |
| Overview | |
| The SLM Vision Parser uses the sequence-to-sequence Florence-2-large model to translate image structures directly. | |
| API Reference | |
| `SLMVisionParser` Initialization | |
| from slm_vision_parser.vision_parser import SLMVisionParser | |
| parser = SLMVisionParser() | |
| [Table Description: The HTML table represents the constructor parameter configuration for SLMVisionParser, detailing the "model_path" parameter of type string (str), which references the local directory containing pre-downloaded checkpoints and defaults to "../../models/florence-2-large".] | |
| `parse_image` Method | |
| Run coordinate mappings or caption generation by supplying task tags. Below is the input flowchart diagram processed by the vision model: | |
| [Image Description: A flowchart showing a start step ('Start Process') and a next step ('Next Step Link') connected with two arrows from the start step to the next step.]</code></pre> | |
| </div> | |
| </section> | |
| <footer style="margin-top: 3rem; text-align: center; border-top: 1px solid #cbd5e1; padding-top: 2rem; color: #475569; font-size: 0.9rem; font-weight: 500;"> | |
| <p>© 2026 SLM Agents. Built with Apache 2.0 Permissive Open Source License.</p> | |
| </footer> | |
| </div> | |
| <!-- GIT CHECKOUT --> | |
| <section class="doc-section" id="git"> | |
| <h2>🐙 Checkout from GitHub</h2> | |
| <p>Clone only this agent's folder from the monorepo using Git sparse-checkout — no need to download the full repository:</p> | |
| <h3 style="font-size: 1.05rem; color: #0f172a; font-weight: 700; margin-top: 1.5rem; margin-bottom: 0.75rem;">Option 1 — Sparse Checkout (Recommended)</h3> | |
| <div class="code-panel" style="max-width:100%; background: #0f172a; border: 1px solid #1e293b; border-radius: 14px; overflow: hidden; margin: 1rem 0; box-shadow: 0 16px 40px rgba(15, 23, 42, 0.12);"> | |
| <div class="code-header" style="background: #1e293b; padding: 10px 16px; display: flex; align-items: center; justify-content: space-between; border-bottom: 1px solid #334155;"> | |
| <div class="code-dots"><div class="code-dot"></div><div class="code-dot"></div><div class="code-dot"></div></div> | |
| <div class="code-title" style="color: #94a3b8; font-weight: 700; font-size: 0.8rem; font-family: 'JetBrains Mono', monospace;">Terminal — Git Sparse Checkout</div> | |
| </div> | |
| <div class="code-content" style="display:block; padding: 1.25rem 1.5rem; background: #0f172a;"> | |
| <pre style="margin:0; background:#0f172a; color:#f8fafc; font-family:'JetBrains Mono',monospace; font-size:0.88rem; border:none; box-shadow:none; padding:0; line-height: 1.75;"><span style="color:#64748b;"># 1. Create and enter a new directory</span> | |
| <span style="color:#34d399;">$</span> <span style="color:#c084fc; font-weight:700;">mkdir</span> <span style="color:#38bdf8;">slm_web_scraper</span> <span style="color:#94a3b8;">&&</span> <span style="color:#c084fc; font-weight:700;">cd</span> <span style="color:#38bdf8;">slm_web_scraper</span> | |
| <span style="color:#64748b;"># 2. Initialise empty git repo and add remote</span> | |
| <span style="color:#34d399;">$</span> <span style="color:#c084fc; font-weight:700;">git init</span> | |
| <span style="color:#34d399;">$</span> <span style="color:#c084fc; font-weight:700;">git remote add origin</span> <span style="color:#38bdf8;">https://github.com/t00114218-stack/SLMAgents.git</span> | |
| <span style="color:#64748b;"># 3. Enable sparse-checkout and set target folder</span> | |
| <span style="color:#34d399;">$</span> <span style="color:#c084fc; font-weight:700;">git sparse-checkout init</span> <span style="color:#94a3b8;">--cone</span> | |
| <span style="color:#34d399;">$</span> <span style="color:#c084fc; font-weight:700;">git sparse-checkout set</span> <span style="color:#38bdf8;">slm_web_scraper</span> | |
| <span style="color:#64748b;"># 4. Pull only that agent's source</span> | |
| <span style="color:#34d399;">$</span> <span style="color:#c084fc; font-weight:700;">git pull origin</span> <span style="color:#38bdf8;">main</span></pre> | |
| </div> | |
| </div> | |
| <h3 style="font-size: 1.05rem; color: #0f172a; font-weight: 700; margin-top: 2rem; margin-bottom: 0.75rem;">Option 2 — Full Repository Clone</h3> | |
| <div class="code-panel" style="max-width:100%; background: #0f172a; border: 1px solid #1e293b; border-radius: 14px; overflow: hidden; margin: 1rem 0;"> | |
| <div class="code-header" style="background: #1e293b; padding: 10px 16px; display: flex; align-items: center; justify-content: space-between; border-bottom: 1px solid #334155;"> | |
| <div class="code-dots"><div class="code-dot"></div><div class="code-dot"></div><div class="code-dot"></div></div> | |
| <div class="code-title" style="color: #94a3b8; font-weight: 700; font-size: 0.8rem; font-family: 'JetBrains Mono', monospace;">Terminal — Full Clone</div> | |
| </div> | |
| <div class="code-content" style="display:block; padding: 1.25rem 1.5rem; background: #0f172a;"> | |
| <pre style="margin:0; background:#0f172a; color:#f8fafc; font-family:'JetBrains Mono',monospace; font-size:0.88rem; border:none; box-shadow:none; padding:0; line-height: 1.75;"><span style="color:#34d399;">$</span> <span style="color:#c084fc; font-weight:700;">git clone</span> <span style="color:#38bdf8;">https://github.com/t00114218-stack/SLMAgents.git</span> | |
| <span style="color:#34d399;">$</span> <span style="color:#c084fc; font-weight:700;">cd</span> <span style="color:#38bdf8;">SLMAgents/slm_web_scraper</span></pre> | |
| </div> | |
| </div> | |
| <p style="margin-top: 1.25rem; font-size: 0.9rem; color: #475569; background: #f8fafc; border: 1px solid #cbd5e1; border-radius: 10px; padding: 1rem 1.25rem;"> | |
| 💡 <strong>Tip:</strong> After checkout, install the package locally with <code style="background: #eef2ff; color: #4f46e5; border: 1px solid #c7d2fe; padding: 2px 8px; border-radius: 5px; font-weight: 700;">pip install -e ./slm_web_scraper</code> to run in editable mode without publishing to PyPI. | |
| </p> | |
| </section> | |
| </main> | |
| </div> | |
| <script src="app.js"></script> | |
| </body> | |
| </html> | |