File size: 7,939 Bytes
82cdf78 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 | ---
title: Internly Backup
emoji: 🚀
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
---
# InternScrapper 🚀 - Student Career Discovery Platform
An intelligence tool for scraping LinkedIn public job listings with both admin and student interfaces. It bypasses auth limits by targeting LinkedIn's guest search interface with fully parameterised filters.
## 🎯 New Features - Student Dashboard
### Admin Interface (/)
- **Job Scraping**: Search and scrape job listings from LinkedIn
- **Real-time Results**: Stream results as they are found
- **School Classification**: Automatically categorize jobs by academic departments
- **Advanced Filters**: Filter by work type, job type, and freshness
- **Persistent Storage**: All scraped jobs are automatically saved to database
### Student Dashboard (/student)
- **Time-based Sections**: View opportunities by when they were discovered
- **Latest Opportunities**: Jobs found in the last hour
- **Yesterday's Opportunities**: Jobs found 1-24 hours ago
- **Auto-refresh**: Updates every 5 minutes
- **Clean Interface**: Focused view for students to browse opportunities
- **24-hour Retention**: Jobs remain visible for 24 hours after discovery
---
## 🏗️ Architecture
InternScrapper uses a **dual-engine architecture** with a fallback policy to ensure highly resilient scrapes, now enhanced with persistent storage.
```
Request ──> [Selenium undetected-chromedriver] ──(Success)──> BeautifulSoup Parser ──> JSON ──> SQLite Database
│ │
(Fail) ▼
▼ Student Dashboard
[Playwright Stealth] ──────────────(Success)──> BeautifulSoup Parser ──> JSON ──> SQLite Database
```
* **Primary Engine:** Selenium + `undetected-chromedriver`. It acts as a standard Google Chrome instance and is highly resilient against Cloudflare, TLS fingerprinting, and standard webdriver checks.
* **Fallback Engine:** Playwright + `playwright-stealth`. If Selenium is not configured or fails, the scrape falls back to Playwright.
* **Unified Parser:** Both engines feed the raw page HTML into a shared `BeautifulSoup4` module utilizing cascading selectors for maximum selector resilience.
* **Persistent Storage:** All scraped jobs are automatically saved to SQLite database with timestamps for the student dashboard.
---
## 🛠️ LinkedIn URL Hacks & Parameter Spec
The tool implements URL query mapping discovered through Reddit OSINT research on guest-job-search parameters:
| Parameter | API Query | Values / Map |
| :--- | :--- | :--- |
| **Experience Level** | `f_E` | `1`=Internship, `2`=Entry Level, `3`=Associate, `4`=Mid-Senior, `5`=Director, `6`=Executive |
| **Job Type** | `f_JT` | `F`=Full-time, `P`=Part-time, `C`=Contract, `T`=Temporary, `I`=Internship, `V`=Volunteer |
| **Work Type** | `f_WT` | `1`=On-site, `2`=Hybrid, `3`=Remote |
| **Posting Age** | `f_TPR` | `r3600`=1 hour, `r14400`=4 hours, `r86400`=24 hours, `r604800`=1 week, `r2592000`=1 month |
---
## ⚙️ Setup & Installation
### Prerequisites
* [Python 3.12+](https://www.python.org/)
* [Node.js 18+](https://nodejs.org/)
* [uv](https://github.com/astral-sh/uv) (Recommended fast package manager)
* Google Chrome installed on your machine (required by undetected-chromedriver)
### 1. Backend Setup (FastAPI)
1. Open a terminal in the root folder (`InternScrapper`):
```bash
# Sync and install all backend dependencies (including setuptools, selenium, playwright, bs4)
uv sync
```
2. Install Playwright browser binaries:
```bash
uv run playwright install chromium
```
3. Start the backend server (without `--reload` to prevent Windows event loop conflicts with Playwright):
```bash
uv run uvicorn main:app
```
The backend will run on [http://localhost:8000](http://localhost:8000).
### 2. Frontend Setup (Next.js)
1. Open a new terminal in the `components` directory:
```bash
cd components
npm install
```
2. Start the development server:
```bash
npm run dev
```
The frontend will run on [http://localhost:3000](http://localhost:3000).
---
## 🖥️ Usage
### For Administrators
1. Navigate to `http://localhost:3000` in your browser.
2. Input your **Keywords** (e.g. `Software Engineer`) and **Location** (e.g. `India` or `United States`).
3. Click **Advanced Filters** to toggle experience levels, job types, work types, and posting freshness.
4. Hit **Scrape**. The UI will display a skeleton loading state, scrape live listings, and display the results including job titles, companies, locations, posting times, and links.
5. All scraped jobs are automatically saved to the database for the student dashboard.
6. Click the **Open this search on LinkedIn ↗** link at any time to verify or review the exact constructed search link directly in your browser.
### For Students
1. Click the **Student Dashboard** button in the admin interface, or navigate to `http://localhost:3000/student`
2. View the **Latest Opportunities** discovered in the last hour
3. Browse **Yesterday's Opportunities** from the past 24 hours
4. Click on any job card to view the full listing on LinkedIn
5. The dashboard auto-refreshes every 5 minutes to show new opportunities
---
## 🚀 Deployment
The project is architected for easy deployment across Vercel (Frontend) and Hugging Face Spaces (Backend).
### 1. Frontend (Vercel)
The Next.js frontend is designed to be hosted on Vercel.
1. Push your repository to GitHub.
2. Import the project in Vercel and set the Root Directory to `components/`.
3. Configure the following Environment Variables in the Vercel Dashboard (see `components/.env.example`):
- `DATABASE_URL` (Neon Postgres URL)
- `BETTER_AUTH_SECRET` (A strong random string)
- `BETTER_AUTH_URL` (Your Vercel production URL, e.g., `https://my-app.vercel.app`)
- `NEXT_PUBLIC_API_URL` (e.g., `https://oh-internscrapper-oppurtunityhub.hf.space`)
- Google OAuth credentials (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`)
### 2. Backend (Hugging Face Spaces)
The Python FastAPI scraper uses Docker to handle heavy dependencies (Chrome/Playwright/Xvfb).
1. Create a Docker Space on Hugging Face.
2. Configure the Space Secrets (Settings > Variables and secrets):
- `DATABASE_URL`
- `SERPER_API_KEY`
3. We have included a GitHub Actions workflow (`.github/workflows/deploy-hf.yml`) that automatically pushes to Hugging Face Spaces when you push to the `main` branch. You just need to add your `HF_TOKEN` as a repository secret in GitHub.
---
## 🗄️ Database & Data Management
### Database Schema
Jobs are stored with:
- Job details (title, company, location, link, posting date)
- Classification (programs, schools)
- Search metadata (keywords, filters used)
- Timestamp (when the job was discovered)
### API Endpoints
#### Admin Endpoints
- `POST /scrape-internships`: Scrape job listings with filters
- `GET /schools`: Get available school/department classifications
- `GET /locations`: Get supported geographic locations
- `POST /admin/cleanup`: Remove old job data
#### Student Endpoints
- `GET /student/jobs/recent?hours={1|24}`: Get jobs from specific timeframe
- `GET /student/jobs/all-timeframes`: Get jobs for both timeframes
### Data Retention
- Jobs remain in the student dashboard for 24 hours after discovery
- Old jobs can be cleaned up using the admin cleanup endpoint
- Database uses SQLite for simplicity and portability
|