orakzai-io commited on
Commit
151bf41
·
verified ·
1 Parent(s): d57e0f9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +52 -44
README.md CHANGED
@@ -1,44 +1,52 @@
1
- # AI-Powered Async Web Scraper
2
-
3
- A professional, high-concurrency web scraping pipeline that extracts clean text from URLs, processes it through an LLM (Groq), and stores the structured results in both JSONL and PostgreSQL.
4
-
5
- ## Quick Start (Docker)
6
-
7
- The easiest way to run the entire stack is using Docker Compose:
8
-
9
- 1. **Configure Environment**:
10
- Create a `.env` file in the `Backend` directory with your Groq API Key:
11
- ```bash
12
- GROQ_API_KEY=your_key_here
13
- DB_PASSWORD=your_db_password
14
- ```
15
-
16
- 2. **Launch**:
17
- ```bash
18
- docker-compose up -d
19
- ```
20
-
21
- 3. **Access**:
22
- - **Frontend**: [http://localhost](http://localhost)
23
- - **Backend API**: [http://localhost:8000](http://localhost:8000)
24
-
25
- ## Architecture
26
-
27
- - **Frontend**: A modern dashboard built with **Vite** and **TypeScript** for real-time job monitoring and result downloads.
28
- - **Backend**: A high-performance **FastAPI** service managing an asynchronous scraping pipeline.
29
- - **Database**: **PostgreSQL** for persistent storage of parsed text and structured AI extractions.
30
- - **Scraper**: Uses `curl_cffi` for advanced request impersonation to bypass bot detection.
31
- - **Parser**: Leverages `trafilatura` for high-quality main-content extraction.
32
- - **Extraction**: Integrates with **Groq** for lightning-fast LLM-based data structuring.
33
-
34
-
35
- ## Project Structure
36
-
37
- - `Backend/`: FastAPI application, database models, and scraper logic.
38
- - `Frontend/`: Vite/TS dashboard for job status tracking and results visualization.
39
- - `docker-compose.yml`: Orchestration for the frontend, backend, and persistence layers.
40
-
41
- ---
42
- For detailed setup and development instructions, see the individual component's documentation:
43
- - [Backend Documentation](./Backend/README.md)
44
- - [Frontend Documentation](./Frontend/README.md)
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ title: Async Web Scraper
4
+ sdk: docker
5
+ emoji: 🦀
6
+ colorFrom: pink
7
+ colorTo: pink
8
+ ---
9
+ # AI-Powered Async Web Scraper
10
+
11
+ A professional, high-concurrency web scraping pipeline that extracts clean text from URLs, processes it through an LLM (Groq), and stores the structured results in both JSONL and PostgreSQL.
12
+
13
+ ## Quick Start (Docker)
14
+
15
+ The easiest way to run the entire stack is using Docker Compose:
16
+
17
+ 1. **Configure Environment**:
18
+ Create a `.env` file in the `Backend` directory with your Groq API Key:
19
+ ```bash
20
+ GROQ_API_KEY=your_key_here
21
+ DB_PASSWORD=your_db_password
22
+ ```
23
+
24
+ 2. **Launch**:
25
+ ```bash
26
+ docker-compose up -d
27
+ ```
28
+
29
+ 3. **Access**:
30
+ - **Frontend**: [http://localhost](http://localhost)
31
+ - **Backend API**: [http://localhost:8000](http://localhost:8000)
32
+
33
+ ## Architecture
34
+
35
+ - **Frontend**: A modern dashboard built with **Vite** and **TypeScript** for real-time job monitoring and result downloads.
36
+ - **Backend**: A high-performance **FastAPI** service managing an asynchronous scraping pipeline.
37
+ - **Database**: **PostgreSQL** for persistent storage of parsed text and structured AI extractions.
38
+ - **Scraper**: Uses `curl_cffi` for advanced request impersonation to bypass bot detection.
39
+ - **Parser**: Leverages `trafilatura` for high-quality main-content extraction.
40
+ - **Extraction**: Integrates with **Groq** for lightning-fast LLM-based data structuring.
41
+
42
+
43
+ ## Project Structure
44
+
45
+ - `Backend/`: FastAPI application, database models, and scraper logic.
46
+ - `Frontend/`: Vite/TS dashboard for job status tracking and results visualization.
47
+ - `docker-compose.yml`: Orchestration for the frontend, backend, and persistence layers.
48
+
49
+ ---
50
+ For detailed setup and development instructions, see the individual component's documentation:
51
+ - [Backend Documentation](./Backend/README.md)
52
+ - [Frontend Documentation](./Frontend/README.md)