--- title: Chorus emoji: 🎧 colorFrom: indigo colorTo: red sdk: gradio sdk_version: 6.16.0 python_version: '3.14' app_file: app.py pinned: false license: mit short_description: Comment filter for signal and insight tags: - track:backyard - sponsor:modal - achievement:offgrid - achievement:offbrand - achievement:llama - achievement:fieldnotes --- * Social media post: https://www.linkedin.com/posts/tijlvercaemer_my-submission-for-the-hugging-face-build-share-7472062466575929345-UrC2/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAMRUHQB6jO6XPWmsbyvEnQfIT-eaQDs0ns * Blog post: https://tijl.blog/posts/build-small-hackathon/ * Demo video: https://www.youtube.com/watch?v=98W_jlyUe4E * Team: tdve # Chorus Chorus is an experimental tool for reading YouTube comment sections. It downloads comments from a YouTube video, filters out low-value comments, clusters the remaining comments, and gives you a summary of the main arguments and observations made by the commenters. The aim is simple: a creator, journalist, writer, or researcher should be able to see what people are really saying without reading thousands of comments by hand. Chorus was built as part of the [Hugging Face Small Model Hackathon](https://huggingface.co/build-small-hackathon). It is not finished software. The current working target is YouTube. Other social networks may be added later. Chorus was *designed to run locally*. Comment fetching, filtering, ranking, clustering, and summarising can all run on your own PC (I'm running it on an AMD Ryzen 9 5950X from 2020). The default setup uses a local [llama.cpp](https://github.com/ggml-org/llama.cpp) server for LLM calls and a local BERTopic embedding model for clustering. You can also point Chorus at a hosted OpenAI-compatible LLM endpoint if you prefer to run the LLM in the cloud. The version of this app which is deployed on Hugging Face, uses an LLM hosted at [Modal.com](https://modal.com) (Thanks to the free credits Modal provided for participants in the Hackaton). A lot of work was done to make it return good enough results within an acceptable amount of time, on hardware without a powerfull GPU. A lot of filtering happens in Python code, before expensive LLM calls are made. ## What Chorus does Given a YouTube video URL, Chorus: 1. Fetches the video comments with the YouTube API. 2. Assigns stable comment IDs. 3. Removes duplicate and near-duplicate comments. 4. Removes comments that look too short, empty, repetitive, or low-information. 5. Builds cleaner text for clustering. 6. Keeps the comments most likely to contain useful information. 7. Clusters similar comments with BERTopic. 8. Selects representative comments for each cluster. 9. Scores clusters with cheap non-LLM rules before using the LLM. 10. Uses an OpenAI-compatible LLM to title and summarise the best clusters. 11. Shows the result in a Gradio web app. The output is a set of comment clusters. Each cluster has a title, a summary, and some representative comments. ## Requirements You need: - Python 3.14, as used by the current Hugging Face Space configuration. - A YouTube Data API key. - A sentence-transformers embedding model for BERTopic. - An OpenAI-compatible LLM endpoint. The LLM endpoint can be local or hosted. The recommended local setup is `llama.cpp` with Qwen3-8B. Install the Python dependencies: ```bash python3 -m venv venv source venv/bin/activate pip install -r requirements.txt ``` ## Running a local LLM with llama.cpp Build or install `llama.cpp`, then start `llama-server`. Recommended model: - `Qwen/Qwen3-8B-GGUF:Q4_K_M` Example command: ```bash ./build/bin/llama-server \ -hf Qwen/Qwen3-8B-GGUF:Q4_K_M \ --host 127.0.0.1 \ --port 8001 \ --ctx-size 4096 \ --alias scorer \ --threads 12 \ --threads-batch 24 \ --batch-size 2048 \ --ubatch-size 512 \ --flash-attn on ``` This exposes an OpenAI-compatible endpoint at: ```text http://127.0.0.1:8001/v1 ``` The model name used by Chorus is the `--alias` value. In the example above, that is: ```text scorer ``` Qwen3-8B is recommended because it is small enough to run locally on many developer machines, but still useful for cluster summaries. Other GGUF models can work if they follow instructions well and return JSON reliably. ## Downloading the BERTopic embedding model Chorus uses BERTopic for clustering. BERTopic needs an embedding model. The recommended model is: ```text BAAI/bge-small-en-v1.5 ``` Download it once and save it locally: ```bash mkdir -p models python - <<'PY' from sentence_transformers import SentenceTransformer model = SentenceTransformer("BAAI/bge-small-en-v1.5") model.save("models/bge-small-en-v1.5") PY ``` Then point Chorus at that local path in `.env`: ```env EMBEDDING_MODEL=./models/bge-small-en-v1.5 EMBEDDING_LOCAL_ONLY=true ``` Use a real path. Do not rely on `~` unless you have checked that the library expands it correctly in your environment. ## Configuration Create a `.env` file in the project root. Minimal local configuration: ```env # YouTube YOUTUBE_API_KEY=your-youtube-data-api-key # BERTopic embeddings EMBEDDING_MODEL=./models/bge-small-en-v1.5 EMBEDDING_LOCAL_ONLY=true # OpenAI-compatible LLM endpoint LLM_BASE_URL=http://127.0.0.1:8001/v1 LLM_MODEL=scorer LLM_API_KEY=sk-no-key-required LLM_TIMEOUT=300 LLM_TEMPERATURE=0 LLM_MAX_TOKENS=512 ``` Useful optional settings: ```env # Cache directory used by the Gradio app CHORUS_CACHE_DIR=.cache/chorus # Number of background jobs the web app may run at the same time CHORUS_MAX_BACKGROUND_JOBS=1 # Number of clusters sent to the LLM after cheap cluster filtering CHORUS_LLM_CLUSTER_BUDGET=8 # Batch several cluster summaries into one LLM call CHORUS_LLM_BATCH_REQUESTS=true CHORUS_LLM_BATCH_CLUSTERS=8 CHORUS_LLM_BATCH_MAX_TOKENS=1200 CHORUS_LLM_MAX_OUTPUT_TOKENS_PER_CLUSTER=220 # Limit how much text from each representative comment is sent to the LLM CHORUS_CLUSTER_COMMENT_CHARS=600 CHORUS_CLUSTER_MAX_REPRESENTATIVES=6 # Pre-clustering comment selection CHORUS_PRECLUSTER_KEEP_FRACTION=0.25 CHORUS_PRECLUSTER_MIN_KEEP=150 CHORUS_PRECLUSTER_MAX_KEEP=1200 CHORUS_PRECLUSTER_MIN_SCORE=4 ``` ## Using a hosted OpenAI-compatible LLM Chorus does not require the LLM to run on the same machine as the web app. Any hosted LLM can be used if it exposes an OpenAI-compatible `/v1/chat/completions` API. This includes a model served from Modal, a private server, or another compatible provider. Set the LLM variables accordingly: ```env LLM_BASE_URL=https://your-hosted-endpoint.example.com/v1 LLM_MODEL=your-model-name LLM_API_KEY=your-api-key-if-required LLM_TIMEOUT=300 LLM_TEMPERATURE=0 LLM_MAX_TOKENS=512 ``` For best results, use a model that can return strict JSON. Chorus asks the LLM for structured cluster titles, summaries, key comment IDs, usefulness scores, and noise scores. ## Running the web app Start the full Gradio app: ```bash python app.py ``` The app accepts a YouTube video URL. It fetches the comments, runs the pipeline, caches the result, and displays the clusters as cards. Cached raw and processed files are stored under the Chorus cache directory. ## Running the scripts directly Fetch comments only: ```bash python fetch_comments.py "https://www.youtube.com/watch?v=VIDEO_ID" > comments.json ``` Run the filtering and clustering pipeline on an existing JSON file: ```bash python filter_comments.py comments.json > output.json ``` The input file should be either: - a JSON list of comments, or - an object with a `comments` field, as produced by `fetch_comments.py`. There is also a helper script for local model testing: ```bash ./run_model_tests.sh ``` That script is intended for development and benchmarking. Check it before running it, because it assumes local files and model settings. ## Filtering pipeline The default pipeline is defined in `chorus/filters/__init__.py`. ### 1. Comment IDs `CommentIDFilter` gives each comment a stable internal ID. Later steps use these IDs when selecting representative comments and when asking the LLM to identify key comments. ### 2. Deduplication `DeduplicationFilter` removes exact and near-duplicate comments. This keeps repeated slogans, copied comments, and minor spelling variants from dominating a cluster. ### 3. Low-information prefilter `LowInformationPrefilter` scores comments with cheap rules. It removes comments that are likely to be too short, too generic, too repetitive, or too empty to help the final summary. Examples of comments this step is meant to reduce include generic praise, one-word reactions, emoji-only comments, and stock fan comments. ### 4. Dynamic clustering text `DynamicClusteringTextFilter` builds the text that will be used for clustering. It normalises text and creates dynamic stopwords from the comment set so that repeated local noise has less influence. ### 5. Top-quality prefilter `TopQualityPrefilter` keeps a limited number of comments before clustering. It combines cheap information score, engagement signals, and other simple measures. This reduces the work done by BERTopic and helps prevent weak comments from setting the topic structure. ### 6. BERTopic clustering `BERTopicClusterFilter` groups similar comments. It uses the embedding model configured by `EMBEDDING_MODEL`. Each cluster receives keywords, a BERTopic topic ID, representative comments, and the comments assigned to that topic. ### 7. Representative comments `RepresentativeCommentsFilter` selects a smaller set of comments from each cluster. These comments are used for display and for LLM summarisation. ### 8. Cluster quality prefilter `ClusterQualityPrefilter` scores each cluster before sending anything to the LLM. Clusters can be marked as: - `send_to_llm` - `keep_unscored` - `drop` This is a cost-control step. It tries to send only the strongest clusters to the LLM. ### 9. Rescue comments `RescueCommentsFilter` looks for strong individual comments that may have landed in a weak cluster. This helps avoid losing useful comments only because their cluster was not good enough. ### 10. LLM cluster scoring and summarising `ScoreClustersFilter` sends the best clusters to the configured LLM. The LLM returns a neutral title, a concise summary, key comment IDs, a usefulness score, and a noise score. Clusters that are kept but not sent to the LLM receive a cheap title and summary based on their keywords. ### 11. Highlight selection `HighlightSelectionFilter` marks the best clusters for display. Highlighted clusters are shown first in the web app. ## Current limits Chorus is experimental. Known limits: - The quality depends heavily on the comment section. - Short comments and joke comments are hard to classify reliably without sending every comment to an LLM. - BERTopic can create odd clusters when the input is small, multilingual, repetitive, or dominated by memes. - The LLM can produce weak summaries if representative comments are poorly chosen. - The current toxicity filter is a placeholder, not a finished moderation system. - The pipeline is tuned by rules and thresholds, not by a trained end-to-end model (filtering out toxic comments was originally the main goal of the project, but was deprioritized when most toxic comments where already removed in other filter steps). - The code still contains some older API code (Reddit), but the supported working target for this README is YouTube. ## Possible improvements ### Better clustering BERTopic is useful, but it is not the only option. Alternatives worth testing: - Agglomerative clustering over sentence embeddings. - HDBSCAN directly over embeddings without the full BERTopic topic model. - K-means or mini-batch k-means for predictable cluster counts. - Two-stage clustering: broad clusters first, then subclusters inside large clusters. - Language-aware clustering for multilingual comment sections. A simpler clustering system may be faster and easier to control than BERTopic. ### Better filtering before clustering The current pre-clustering filters are cheap and rule-based. They could be improved by adding a small LLM or classifier before clustering. Possible additions: - A small local model that scores each comment for insight, specificity, criticism, question, correction, argument, and noise. - A separate classifier for spam, abuse, and generic fan comments. - Better handling of replies, so reply chains are not treated as isolated comments. - Better detection of copied comments and repeated slogans. - Stronger multilingual handling (Clustering in other languages already works fairly well, but a lot of the cheap filters rely on hard-coded English keywords) This stage matters because clustering only works well if the input comments are worth clustering. ### Better filtering after clustering The current cluster quality step uses cheap rules before LLM scoring. It could be made stronger. Possible additions: - Penalise clusters where the representative comments do not agree with the keywords. - Detect clusters that are mostly jokes, memes, insults, or meta-discussion. - Split clusters that contain several unrelated arguments. - Merge clusters that BERTopic separated but the LLM sees as the same argument. - Use a second pass to produce a final ranked list of the most useful points. ### Better summaries The LLM summary step could be improved by changing what is sent to the model. Possible additions: - Include more structured metadata about each comment. - Send a better spread of comments from each cluster, not only the top representatives. - Ask for separate fields: main claim, supporting reasons, criticism, evidence, suggested action, and disagreement. - Ask the LLM to state when a cluster is too noisy to summarise. - Add a final cross-cluster summary that removes duplication between clusters. ### Better UI The current app shows cluster cards. Useful additions would be: - Filters for highlighted, unscored, noisy, or low-confidence clusters. - Sorting by usefulness, size, likes, replies, or noise. - Export to Markdown, JSON, or CSV. - A view that shows only the best comments across all clusters. - A way to mark bad clusters and use that feedback to tune thresholds. ### Performance work The slowest part is usually LLM scoring. There is a strong tradeoff between performance and quality. That balance is currently already tilting a bit too much towards prioritizing performance. ### Small tuning changes Some useful small changes: - Tune `CHORUS_PRECLUSTER_KEEP_FRACTION` per comment volume. - Tune `CHORUS_LLM_CLUSTER_BUDGET` for speed versus summary coverage. - Lower `CHORUS_CLUSTER_COMMENT_CHARS` if LLM calls are too slow. - Raise `CHORUS_PRECLUSTER_MIN_SCORE` for noisy comment sections. - Keep more comments before clustering when comment sections are small. - Add tests with comment threads that are multilingual, toxic, repetitive, or very large. ## License This project is licensed under the MIT License. See `LICENSE` for details.