Hugging Face's logo

Spaces:
OH-InternScrapper
/
Internly-Backup
Paused

App Files Community
Internly-Backup / README.md
Sambhavvvvv's picture
Sambhavvvvv
Upload README.md
82cdf78 verified
preview code
|
Raw
History Blame Contribute Delete
7.94 kB
metadata 
title: Internly Backup
emoji: 🚀
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false

InternScrapper 🚀 - Student Career Discovery Platform

An intelligence tool for scraping LinkedIn public job listings with both admin and student interfaces. It bypasses auth limits by targeting LinkedIn's guest search interface with fully parameterised filters.

🎯 New Features - Student Dashboard

Admin Interface (/)

  • Job Scraping: Search and scrape job listings from LinkedIn
  • Real-time Results: Stream results as they are found
  • School Classification: Automatically categorize jobs by academic departments
  • Advanced Filters: Filter by work type, job type, and freshness
  • Persistent Storage: All scraped jobs are automatically saved to database

Student Dashboard (/student)

  • Time-based Sections: View opportunities by when they were discovered
    • Latest Opportunities: Jobs found in the last hour
    • Yesterday's Opportunities: Jobs found 1-24 hours ago
  • Auto-refresh: Updates every 5 minutes
  • Clean Interface: Focused view for students to browse opportunities
  • 24-hour Retention: Jobs remain visible for 24 hours after discovery

🏗️ Architecture

InternScrapper uses a dual-engine architecture with a fallback policy to ensure highly resilient scrapes, now enhanced with persistent storage.

Request ──> [Selenium undetected-chromedriver] ──(Success)──> BeautifulSoup Parser ──> JSON ──> SQLite Database
                    │                                                                      │
                 (Fail)                                                                    ▼
                    ▼                                                              Student Dashboard
            [Playwright Stealth] ──────────────(Success)──> BeautifulSoup Parser ──> JSON ──> SQLite Database
  • Primary Engine: Selenium + undetected-chromedriver. It acts as a standard Google Chrome instance and is highly resilient against Cloudflare, TLS fingerprinting, and standard webdriver checks.
  • Fallback Engine: Playwright + playwright-stealth. If Selenium is not configured or fails, the scrape falls back to Playwright.
  • Unified Parser: Both engines feed the raw page HTML into a shared BeautifulSoup4 module utilizing cascading selectors for maximum selector resilience.
  • Persistent Storage: All scraped jobs are automatically saved to SQLite database with timestamps for the student dashboard.

🛠️ LinkedIn URL Hacks & Parameter Spec

The tool implements URL query mapping discovered through Reddit OSINT research on guest-job-search parameters:

Parameter API Query Values / Map
Experience Level f_E 1=Internship, 2=Entry Level, 3=Associate, 4=Mid-Senior, 5=Director, 6=Executive
Job Type f_JT F=Full-time, P=Part-time, C=Contract, T=Temporary, I=Internship, V=Volunteer
Work Type f_WT 1=On-site, 2=Hybrid, 3=Remote
Posting Age f_TPR r3600=1 hour, r14400=4 hours, r86400=24 hours, r604800=1 week, r2592000=1 month

⚙️ Setup & Installation

Prerequisites

  • Python 3.12+
  • Node.js 18+
  • uv (Recommended fast package manager)
  • Google Chrome installed on your machine (required by undetected-chromedriver)

1. Backend Setup (FastAPI)

  1. Open a terminal in the root folder (InternScrapper):
    # Sync and install all backend dependencies (including setuptools, selenium, playwright, bs4)
uv sync
  2. Install Playwright browser binaries:
    uv run playwright install chromium
  3. Start the backend server (without --reload to prevent Windows event loop conflicts with Playwright):
    uv run uvicorn main:app
    The backend will run on http://localhost:8000.

2. Frontend Setup (Next.js)

  1. Open a new terminal in the components directory:
    cd components
npm install
  2. Start the development server:
    npm run dev
    The frontend will run on http://localhost:3000.

🖥️ Usage

For Administrators

  1. Navigate to http://localhost:3000 in your browser.
  2. Input your Keywords (e.g. Software Engineer) and Location (e.g. India or United States).
  3. Click Advanced Filters to toggle experience levels, job types, work types, and posting freshness.
  4. Hit Scrape. The UI will display a skeleton loading state, scrape live listings, and display the results including job titles, companies, locations, posting times, and links.
  5. All scraped jobs are automatically saved to the database for the student dashboard.
  6. Click the Open this search on LinkedIn ↗ link at any time to verify or review the exact constructed search link directly in your browser.

For Students

  1. Click the Student Dashboard button in the admin interface, or navigate to http://localhost:3000/student
  2. View the Latest Opportunities discovered in the last hour
  3. Browse Yesterday's Opportunities from the past 24 hours
  4. Click on any job card to view the full listing on LinkedIn
  5. The dashboard auto-refreshes every 5 minutes to show new opportunities

🚀 Deployment

The project is architected for easy deployment across Vercel (Frontend) and Hugging Face Spaces (Backend).

1. Frontend (Vercel)

The Next.js frontend is designed to be hosted on Vercel.

  1. Push your repository to GitHub.
  2. Import the project in Vercel and set the Root Directory to components/.
  3. Configure the following Environment Variables in the Vercel Dashboard (see components/.env.example):
    • DATABASE_URL (Neon Postgres URL)
    • BETTER_AUTH_SECRET (A strong random string)
    • BETTER_AUTH_URL (Your Vercel production URL, e.g., https://my-app.vercel.app)
    • NEXT_PUBLIC_API_URL (e.g., https://oh-internscrapper-oppurtunityhub.hf.space)
    • Google OAuth credentials (GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET)

2. Backend (Hugging Face Spaces)

The Python FastAPI scraper uses Docker to handle heavy dependencies (Chrome/Playwright/Xvfb).

  1. Create a Docker Space on Hugging Face.
  2. Configure the Space Secrets (Settings > Variables and secrets):
    • DATABASE_URL
    • SERPER_API_KEY
  3. We have included a GitHub Actions workflow (.github/workflows/deploy-hf.yml) that automatically pushes to Hugging Face Spaces when you push to the main branch. You just need to add your HF_TOKEN as a repository secret in GitHub.

🗄️ Database & Data Management

Database Schema

Jobs are stored with:

  • Job details (title, company, location, link, posting date)
  • Classification (programs, schools)
  • Search metadata (keywords, filters used)
  • Timestamp (when the job was discovered)

API Endpoints

Admin Endpoints

  • POST /scrape-internships: Scrape job listings with filters
  • GET /schools: Get available school/department classifications
  • GET /locations: Get supported geographic locations
  • POST /admin/cleanup: Remove old job data

Student Endpoints

  • GET /student/jobs/recent?hours={1|24}: Get jobs from specific timeframe
  • GET /student/jobs/all-timeframes: Get jobs for both timeframes

Data Retention

  • Jobs remain in the student dashboard for 24 hours after discovery
  • Old jobs can be cleaned up using the admin cleanup endpoint
  • Database uses SQLite for simplicity and portability