| --- |
| title: Reddit SemanticSearch Prototype |
| emoji: 🐨 |
| colorFrom: purple |
| colorTo: indigo |
| sdk: gradio |
| sdk_version: 5.41.0 |
| app_file: app.py |
| pinned: false |
| short_description: 'r/technology, r/gaming, r/programming etc search comments ' |
| --- |
| |
| # Reddit Semantic Search (Prototype) |
|
|
| A lightweight semantic search engine built on Reddit comments using: |
| - **Word2Vec embeddings** (trained from scratch on selected subreddits) |
| - **FAISS** for fast vector indexing and retrieval |
| - **Gradio** for a user-friendly, Reddit-themed interface |
|
|
| > ⚠️ This is an independent prototype. Not affiliated with Reddit Inc. |
|
|
| --- |
|
|
| ## Dataset |
|
|
| - Source: [`HuggingFaceGECLM/REDDIT_comments`](https://huggingface.co/datasets/HuggingFaceGECLM/REDDIT_comments) |
| - Subreddits used: |
| - `askscience`, `gaming`, `technology`, `todayilearned`, `programming` |
| - Data was streamed using Hugging Face's `datasets` library and chunked using PySpark. |
|
|
| --- |
|
|
| ## Project Pipeline |
|
|
| 1. **Data Loading & Chunking** |
| - Load subreddit splits individually using streaming |
| - Group every 5 comments into a single text chunk using PySpark |
| - Clean and tokenize text for training |
|
|
| 2. **Training Word2Vec** |
| - Custom embeddings trained using `gensim`'s Word2Vec on cleaned comment chunks |
|
|
| 3. **Vector Indexing (FAISS)** |
| - Each chunk embedded by averaging Word2Vec vectors of words |
| - Dense vectors indexed using `faiss.IndexFlatL2` |
|
|
| 4. **Semantic Search App (Gradio)** |
| - Enter your query and select a subreddit filter |
| - Retrieves top 5 semantically similar comment chunks |
| - Built-in reranking logic can be added later |
|
|
| --- |
|
|
| ## Run the App |
|
|
| ```bash |
| pip install -r requirements.txt |
| python app.py # or run the notebook |
| |
| |
| Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference |
| |
