Spaces:
Sleeping
Sleeping
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,44 +1,52 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
```
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
- **
|
| 31 |
-
- **
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
-
|
| 38 |
-
-
|
| 39 |
-
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
title: Async Web Scraper
|
| 4 |
+
sdk: docker
|
| 5 |
+
emoji: 🦀
|
| 6 |
+
colorFrom: pink
|
| 7 |
+
colorTo: pink
|
| 8 |
+
---
|
| 9 |
+
# AI-Powered Async Web Scraper
|
| 10 |
+
|
| 11 |
+
A professional, high-concurrency web scraping pipeline that extracts clean text from URLs, processes it through an LLM (Groq), and stores the structured results in both JSONL and PostgreSQL.
|
| 12 |
+
|
| 13 |
+
## Quick Start (Docker)
|
| 14 |
+
|
| 15 |
+
The easiest way to run the entire stack is using Docker Compose:
|
| 16 |
+
|
| 17 |
+
1. **Configure Environment**:
|
| 18 |
+
Create a `.env` file in the `Backend` directory with your Groq API Key:
|
| 19 |
+
```bash
|
| 20 |
+
GROQ_API_KEY=your_key_here
|
| 21 |
+
DB_PASSWORD=your_db_password
|
| 22 |
+
```
|
| 23 |
+
|
| 24 |
+
2. **Launch**:
|
| 25 |
+
```bash
|
| 26 |
+
docker-compose up -d
|
| 27 |
+
```
|
| 28 |
+
|
| 29 |
+
3. **Access**:
|
| 30 |
+
- **Frontend**: [http://localhost](http://localhost)
|
| 31 |
+
- **Backend API**: [http://localhost:8000](http://localhost:8000)
|
| 32 |
+
|
| 33 |
+
## Architecture
|
| 34 |
+
|
| 35 |
+
- **Frontend**: A modern dashboard built with **Vite** and **TypeScript** for real-time job monitoring and result downloads.
|
| 36 |
+
- **Backend**: A high-performance **FastAPI** service managing an asynchronous scraping pipeline.
|
| 37 |
+
- **Database**: **PostgreSQL** for persistent storage of parsed text and structured AI extractions.
|
| 38 |
+
- **Scraper**: Uses `curl_cffi` for advanced request impersonation to bypass bot detection.
|
| 39 |
+
- **Parser**: Leverages `trafilatura` for high-quality main-content extraction.
|
| 40 |
+
- **Extraction**: Integrates with **Groq** for lightning-fast LLM-based data structuring.
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
## Project Structure
|
| 44 |
+
|
| 45 |
+
- `Backend/`: FastAPI application, database models, and scraper logic.
|
| 46 |
+
- `Frontend/`: Vite/TS dashboard for job status tracking and results visualization.
|
| 47 |
+
- `docker-compose.yml`: Orchestration for the frontend, backend, and persistence layers.
|
| 48 |
+
|
| 49 |
+
---
|
| 50 |
+
For detailed setup and development instructions, see the individual component's documentation:
|
| 51 |
+
- [Backend Documentation](./Backend/README.md)
|
| 52 |
+
- [Frontend Documentation](./Frontend/README.md)
|