| ---
|
| title: Internly Backup
|
| emoji: 🚀
|
| colorFrom: blue
|
| colorTo: indigo
|
| sdk: docker
|
| app_port: 7860
|
| pinned: false
|
| ---
|
|
|
| # InternScrapper 🚀 - Student Career Discovery Platform
|
|
|
| An intelligence tool for scraping LinkedIn public job listings with both admin and student interfaces. It bypasses auth limits by targeting LinkedIn's guest search interface with fully parameterised filters.
|
|
|
| ## 🎯 New Features - Student Dashboard
|
|
|
| ### Admin Interface (/)
|
| - **Job Scraping**: Search and scrape job listings from LinkedIn
|
| - **Real-time Results**: Stream results as they are found
|
| - **School Classification**: Automatically categorize jobs by academic departments
|
| - **Advanced Filters**: Filter by work type, job type, and freshness
|
| - **Persistent Storage**: All scraped jobs are automatically saved to database
|
|
|
| ### Student Dashboard (/student)
|
| - **Time-based Sections**: View opportunities by when they were discovered
|
| - **Latest Opportunities**: Jobs found in the last hour
|
| - **Yesterday's Opportunities**: Jobs found 1-24 hours ago
|
| - **Auto-refresh**: Updates every 5 minutes
|
| - **Clean Interface**: Focused view for students to browse opportunities
|
| - **24-hour Retention**: Jobs remain visible for 24 hours after discovery
|
|
|
| ---
|
|
|
| ## 🏗️ Architecture
|
|
|
| InternScrapper uses a **dual-engine architecture** with a fallback policy to ensure highly resilient scrapes, now enhanced with persistent storage.
|
|
|
| ```
|
| Request ──> [Selenium undetected-chromedriver] ──(Success)──> BeautifulSoup Parser ──> JSON ──> SQLite Database
|
| │ │
|
| (Fail) ▼
|
| ▼ Student Dashboard
|
| [Playwright Stealth] ──────────────(Success)──> BeautifulSoup Parser ──> JSON ──> SQLite Database
|
| ```
|
|
|
| * **Primary Engine:** Selenium + `undetected-chromedriver`. It acts as a standard Google Chrome instance and is highly resilient against Cloudflare, TLS fingerprinting, and standard webdriver checks.
|
| * **Fallback Engine:** Playwright + `playwright-stealth`. If Selenium is not configured or fails, the scrape falls back to Playwright.
|
| * **Unified Parser:** Both engines feed the raw page HTML into a shared `BeautifulSoup4` module utilizing cascading selectors for maximum selector resilience.
|
| * **Persistent Storage:** All scraped jobs are automatically saved to SQLite database with timestamps for the student dashboard.
|
|
|
| ---
|
|
|
| ## 🛠️ LinkedIn URL Hacks & Parameter Spec
|
|
|
| The tool implements URL query mapping discovered through Reddit OSINT research on guest-job-search parameters:
|
|
|
| | Parameter | API Query | Values / Map |
|
| | :--- | :--- | :--- |
|
| | **Experience Level** | `f_E` | `1`=Internship, `2`=Entry Level, `3`=Associate, `4`=Mid-Senior, `5`=Director, `6`=Executive |
|
| | **Job Type** | `f_JT` | `F`=Full-time, `P`=Part-time, `C`=Contract, `T`=Temporary, `I`=Internship, `V`=Volunteer |
|
| | **Work Type** | `f_WT` | `1`=On-site, `2`=Hybrid, `3`=Remote |
|
| | **Posting Age** | `f_TPR` | `r3600`=1 hour, `r14400`=4 hours, `r86400`=24 hours, `r604800`=1 week, `r2592000`=1 month |
|
|
|
| ---
|
|
|
| ## ⚙️ Setup & Installation
|
|
|
| ### Prerequisites
|
| * [Python 3.12+](https://www.python.org/)
|
| * [Node.js 18+](https://nodejs.org/)
|
| * [uv](https://github.com/astral-sh/uv) (Recommended fast package manager)
|
| * Google Chrome installed on your machine (required by undetected-chromedriver)
|
|
|
| ### 1. Backend Setup (FastAPI)
|
|
|
| 1. Open a terminal in the root folder (`InternScrapper`):
|
| ```bash
|
| # Sync and install all backend dependencies (including setuptools, selenium, playwright, bs4)
|
| uv sync
|
| ```
|
| 2. Install Playwright browser binaries:
|
| ```bash
|
| uv run playwright install chromium
|
| ```
|
| 3. Start the backend server (without `--reload` to prevent Windows event loop conflicts with Playwright):
|
| ```bash
|
| uv run uvicorn main:app
|
| ```
|
| The backend will run on [http://localhost:8000](http://localhost:8000).
|
|
|
| ### 2. Frontend Setup (Next.js)
|
|
|
| 1. Open a new terminal in the `components` directory:
|
| ```bash
|
| cd components
|
| npm install
|
| ```
|
| 2. Start the development server:
|
| ```bash
|
| npm run dev
|
| ```
|
| The frontend will run on [http://localhost:3000](http://localhost:3000).
|
|
|
| ---
|
|
|
| ## 🖥️ Usage
|
|
|
| ### For Administrators
|
| 1. Navigate to `http://localhost:3000` in your browser.
|
| 2. Input your **Keywords** (e.g. `Software Engineer`) and **Location** (e.g. `India` or `United States`).
|
| 3. Click **Advanced Filters** to toggle experience levels, job types, work types, and posting freshness.
|
| 4. Hit **Scrape**. The UI will display a skeleton loading state, scrape live listings, and display the results including job titles, companies, locations, posting times, and links.
|
| 5. All scraped jobs are automatically saved to the database for the student dashboard.
|
| 6. Click the **Open this search on LinkedIn ↗** link at any time to verify or review the exact constructed search link directly in your browser.
|
|
|
| ### For Students
|
| 1. Click the **Student Dashboard** button in the admin interface, or navigate to `http://localhost:3000/student`
|
| 2. View the **Latest Opportunities** discovered in the last hour
|
| 3. Browse **Yesterday's Opportunities** from the past 24 hours
|
| 4. Click on any job card to view the full listing on LinkedIn
|
| 5. The dashboard auto-refreshes every 5 minutes to show new opportunities
|
|
|
| ---
|
|
|
| ## 🚀 Deployment
|
|
|
| The project is architected for easy deployment across Vercel (Frontend) and Hugging Face Spaces (Backend).
|
|
|
| ### 1. Frontend (Vercel)
|
| The Next.js frontend is designed to be hosted on Vercel.
|
| 1. Push your repository to GitHub.
|
| 2. Import the project in Vercel and set the Root Directory to `components/`.
|
| 3. Configure the following Environment Variables in the Vercel Dashboard (see `components/.env.example`):
|
| - `DATABASE_URL` (Neon Postgres URL)
|
| - `BETTER_AUTH_SECRET` (A strong random string)
|
| - `BETTER_AUTH_URL` (Your Vercel production URL, e.g., `https://my-app.vercel.app`)
|
| - `NEXT_PUBLIC_API_URL` (e.g., `https://oh-internscrapper-oppurtunityhub.hf.space`)
|
| - Google OAuth credentials (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`)
|
|
|
| ### 2. Backend (Hugging Face Spaces)
|
| The Python FastAPI scraper uses Docker to handle heavy dependencies (Chrome/Playwright/Xvfb).
|
| 1. Create a Docker Space on Hugging Face.
|
| 2. Configure the Space Secrets (Settings > Variables and secrets):
|
| - `DATABASE_URL`
|
| - `SERPER_API_KEY`
|
| 3. We have included a GitHub Actions workflow (`.github/workflows/deploy-hf.yml`) that automatically pushes to Hugging Face Spaces when you push to the `main` branch. You just need to add your `HF_TOKEN` as a repository secret in GitHub.
|
|
|
| ---
|
|
|
| ## 🗄️ Database & Data Management
|
|
|
| ### Database Schema
|
| Jobs are stored with:
|
| - Job details (title, company, location, link, posting date)
|
| - Classification (programs, schools)
|
| - Search metadata (keywords, filters used)
|
| - Timestamp (when the job was discovered)
|
|
|
| ### API Endpoints
|
|
|
| #### Admin Endpoints
|
| - `POST /scrape-internships`: Scrape job listings with filters
|
| - `GET /schools`: Get available school/department classifications
|
| - `GET /locations`: Get supported geographic locations
|
| - `POST /admin/cleanup`: Remove old job data
|
|
|
| #### Student Endpoints
|
| - `GET /student/jobs/recent?hours={1|24}`: Get jobs from specific timeframe
|
| - `GET /student/jobs/all-timeframes`: Get jobs for both timeframes
|
|
|
| ### Data Retention
|
| - Jobs remain in the student dashboard for 24 hours after discovery
|
| - Old jobs can be cleaned up using the admin cleanup endpoint
|
| - Database uses SQLite for simplicity and portability
|
|
|
