README.md

---
title: Chorus
emoji: 🎧
colorFrom: indigo
colorTo: red
sdk: gradio
sdk_version: 6.16.0
python_version: '3.14'
app_file: app.py
pinned: false
license: mit
short_description: Comment filter for signal and insight
tags:
  - track:backyard
  - sponsor:modal
  - achievement:offgrid
  - achievement:offbrand
  - achievement:llama
  - achievement:fieldnotes
---

* Social media post: https://www.linkedin.com/posts/tijlvercaemer_my-submission-for-the-hugging-face-build-share-7472062466575929345-UrC2/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAMRUHQB6jO6XPWmsbyvEnQfIT-eaQDs0ns
* Blog post: https://tijl.blog/posts/build-small-hackathon/
* Demo video: https://www.youtube.com/watch?v=98W_jlyUe4E
* Team: tdve

# Chorus

Chorus is an experimental tool for reading YouTube comment sections.

It downloads comments from a YouTube video, filters out low-value comments, clusters the remaining comments, and gives you a summary of the main arguments and observations made by the commenters.

The aim is simple: a creator, journalist, writer, or researcher should be able to see what people are really saying without reading thousands of comments by hand.

Chorus was built as part of the [Hugging Face Small Model Hackathon](https://huggingface.co/build-small-hackathon). It is not finished software. The current working target is YouTube. Other social networks may be added later.

Chorus was *designed to run locally*. Comment fetching, filtering, ranking, clustering, and summarising can all run on your own PC (I'm running it on an AMD Ryzen 9 5950X from 2020). The default setup uses a local [llama.cpp](https://github.com/ggml-org/llama.cpp) server for LLM calls and a local BERTopic embedding model for clustering. You can also point Chorus at a hosted OpenAI-compatible LLM endpoint if you prefer to run the LLM in the cloud. The version of this app which is deployed on Hugging Face, uses an LLM hosted at [Modal.com](https://modal.com) (Thanks to the free credits Modal provided for participants in the Hackaton). A lot of work was done to make it return good enough results within an acceptable amount of time, on hardware without a powerfull GPU. A lot of filtering happens in Python code, before expensive LLM calls are made.

## What Chorus does

Given a YouTube video URL, Chorus:

1. Fetches the video comments with the YouTube API.
2. Assigns stable comment IDs.
3. Removes duplicate and near-duplicate comments.
4. Removes comments that look too short, empty, repetitive, or low-information.
5. Builds cleaner text for clustering.
6. Keeps the comments most likely to contain useful information.
7. Clusters similar comments with BERTopic.
8. Selects representative comments for each cluster.
9. Scores clusters with cheap non-LLM rules before using the LLM.
10. Uses an OpenAI-compatible LLM to title and summarise the best clusters.
11. Shows the result in a Gradio web app.

The output is a set of comment clusters. Each cluster has a title, a summary, and some representative comments.

## Requirements

You need:

- Python 3.14, as used by the current Hugging Face Space configuration.
- A YouTube Data API key.
- A sentence-transformers embedding model for BERTopic.
- An OpenAI-compatible LLM endpoint.

The LLM endpoint can be local or hosted. The recommended local setup is `llama.cpp` with Qwen3-8B.

Install the Python dependencies:

```bash
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```

## Running a local LLM with llama.cpp

Build or install `llama.cpp`, then start `llama-server`.

Recommended model:

- `Qwen/Qwen3-8B-GGUF:Q4_K_M`

Example command:

```bash
./build/bin/llama-server \
  -hf Qwen/Qwen3-8B-GGUF:Q4_K_M \
  --host 127.0.0.1 \
  --port 8001 \
  --ctx-size 4096 \
  --alias scorer \
  --threads 12 \
  --threads-batch 24 \
  --batch-size 2048 \
  --ubatch-size 512 \
  --flash-attn on
```

This exposes an OpenAI-compatible endpoint at:

```text
http://127.0.0.1:8001/v1
```

The model name used by Chorus is the `--alias` value. In the example above, that is:

```text
scorer
```

Qwen3-8B is recommended because it is small enough to run locally on many developer machines, but still useful for cluster summaries. Other GGUF models can work if they follow instructions well and return JSON reliably.

## Downloading the BERTopic embedding model

Chorus uses BERTopic for clustering. BERTopic needs an embedding model. The recommended model is:

```text
BAAI/bge-small-en-v1.5
```

Download it once and save it locally:

```bash
mkdir -p models
python - <<'PY'
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
model.save("models/bge-small-en-v1.5")
PY
```

Then point Chorus at that local path in `.env`:

```env
EMBEDDING_MODEL=./models/bge-small-en-v1.5
EMBEDDING_LOCAL_ONLY=true
```

Use a real path. Do not rely on `~` unless you have checked that the library expands it correctly in your environment.

## Configuration

Create a `.env` file in the project root.

Minimal local configuration:

```env
# YouTube
YOUTUBE_API_KEY=your-youtube-data-api-key

# BERTopic embeddings
EMBEDDING_MODEL=./models/bge-small-en-v1.5
EMBEDDING_LOCAL_ONLY=true

# OpenAI-compatible LLM endpoint
LLM_BASE_URL=http://127.0.0.1:8001/v1
LLM_MODEL=scorer
LLM_API_KEY=sk-no-key-required
LLM_TIMEOUT=300
LLM_TEMPERATURE=0
LLM_MAX_TOKENS=512
```

Useful optional settings:

```env
# Cache directory used by the Gradio app
CHORUS_CACHE_DIR=.cache/chorus

# Number of background jobs the web app may run at the same time
CHORUS_MAX_BACKGROUND_JOBS=1

# Number of clusters sent to the LLM after cheap cluster filtering
CHORUS_LLM_CLUSTER_BUDGET=8

# Batch several cluster summaries into one LLM call
CHORUS_LLM_BATCH_REQUESTS=true
CHORUS_LLM_BATCH_CLUSTERS=8
CHORUS_LLM_BATCH_MAX_TOKENS=1200
CHORUS_LLM_MAX_OUTPUT_TOKENS_PER_CLUSTER=220

# Limit how much text from each representative comment is sent to the LLM
CHORUS_CLUSTER_COMMENT_CHARS=600
CHORUS_CLUSTER_MAX_REPRESENTATIVES=6

# Pre-clustering comment selection
CHORUS_PRECLUSTER_KEEP_FRACTION=0.25
CHORUS_PRECLUSTER_MIN_KEEP=150
CHORUS_PRECLUSTER_MAX_KEEP=1200
CHORUS_PRECLUSTER_MIN_SCORE=4
```

## Using a hosted OpenAI-compatible LLM

Chorus does not require the LLM to run on the same machine as the web app.

Any hosted LLM can be used if it exposes an OpenAI-compatible `/v1/chat/completions` API. This includes a model served from Modal, a private server, or another compatible provider.

Set the LLM variables accordingly:

```env
LLM_BASE_URL=https://your-hosted-endpoint.example.com/v1
LLM_MODEL=your-model-name
LLM_API_KEY=your-api-key-if-required
LLM_TIMEOUT=300
LLM_TEMPERATURE=0
LLM_MAX_TOKENS=512
```

For best results, use a model that can return strict JSON. Chorus asks the LLM for structured cluster titles, summaries, key comment IDs, usefulness scores, and noise scores.

## Running the web app

Start the full Gradio app:

```bash
python app.py
```

The app accepts a YouTube video URL. It fetches the comments, runs the pipeline, caches the result, and displays the clusters as cards. Cached raw and processed files are stored under the Chorus cache directory.

## Running the scripts directly

Fetch comments only:

```bash
python fetch_comments.py "https://www.youtube.com/watch?v=VIDEO_ID" > comments.json
```

Run the filtering and clustering pipeline on an existing JSON file:

```bash
python filter_comments.py comments.json > output.json
```

The input file should be either:

- a JSON list of comments, or
- an object with a `comments` field, as produced by `fetch_comments.py`.

There is also a helper script for local model testing:

```bash
./run_model_tests.sh
```

That script is intended for development and benchmarking. Check it before running it, because it assumes local files and model settings.

## Filtering pipeline

The default pipeline is defined in `chorus/filters/__init__.py`.

### 1. Comment IDs

`CommentIDFilter` gives each comment a stable internal ID. Later steps use these IDs when selecting representative comments and when asking the LLM to identify key comments.

### 2. Deduplication

`DeduplicationFilter` removes exact and near-duplicate comments. This keeps repeated slogans, copied comments, and minor spelling variants from dominating a cluster.

### 3. Low-information prefilter

`LowInformationPrefilter` scores comments with cheap rules. It removes comments that are likely to be too short, too generic, too repetitive, or too empty to help the final summary.

Examples of comments this step is meant to reduce include generic praise, one-word reactions, emoji-only comments, and stock fan comments.

### 4. Dynamic clustering text

`DynamicClusteringTextFilter` builds the text that will be used for clustering. It normalises text and creates dynamic stopwords from the comment set so that repeated local noise has less influence.

### 5. Top-quality prefilter

`TopQualityPrefilter` keeps a limited number of comments before clustering. It combines cheap information score, engagement signals, and other simple measures. This reduces the work done by BERTopic and helps prevent weak comments from setting the topic structure.

### 6. BERTopic clustering

`BERTopicClusterFilter` groups similar comments. It uses the embedding model configured by `EMBEDDING_MODEL`. Each cluster receives keywords, a BERTopic topic ID, representative comments, and the comments assigned to that topic.

### 7. Representative comments

`RepresentativeCommentsFilter` selects a smaller set of comments from each cluster. These comments are used for display and for LLM summarisation.

### 8. Cluster quality prefilter

`ClusterQualityPrefilter` scores each cluster before sending anything to the LLM. Clusters can be marked as:

- `send_to_llm`
- `keep_unscored`
- `drop`

This is a cost-control step. It tries to send only the strongest clusters to the LLM.

### 9. Rescue comments

`RescueCommentsFilter` looks for strong individual comments that may have landed in a weak cluster. This helps avoid losing useful comments only because their cluster was not good enough.

### 10. LLM cluster scoring and summarising

`ScoreClustersFilter` sends the best clusters to the configured LLM. The LLM returns a neutral title, a concise summary, key comment IDs, a usefulness score, and a noise score.

Clusters that are kept but not sent to the LLM receive a cheap title and summary based on their keywords.

### 11. Highlight selection

`HighlightSelectionFilter` marks the best clusters for display. Highlighted clusters are shown first in the web app.

## Current limits

Chorus is experimental.

Known limits:

- The quality depends heavily on the comment section.
- Short comments and joke comments are hard to classify reliably without sending every comment to an LLM.
- BERTopic can create odd clusters when the input is small, multilingual, repetitive, or dominated by memes.
- The LLM can produce weak summaries if representative comments are poorly chosen.
- The current toxicity filter is a placeholder, not a finished moderation system.
- The pipeline is tuned by rules and thresholds, not by a trained end-to-end model (filtering out toxic comments was originally the main goal of the project, but was deprioritized when most toxic comments where already removed in other filter steps).
- The code still contains some older API code (Reddit), but the supported working target for this README is YouTube.

## Possible improvements

### Better clustering

BERTopic is useful, but it is not the only option. Alternatives worth testing:

- Agglomerative clustering over sentence embeddings.
- HDBSCAN directly over embeddings without the full BERTopic topic model.
- K-means or mini-batch k-means for predictable cluster counts.
- Two-stage clustering: broad clusters first, then subclusters inside large clusters.
- Language-aware clustering for multilingual comment sections.

A simpler clustering system may be faster and easier to control than BERTopic.

### Better filtering before clustering

The current pre-clustering filters are cheap and rule-based. They could be improved by adding a small LLM or classifier before clustering.

Possible additions:

- A small local model that scores each comment for insight, specificity, criticism, question, correction, argument, and noise.
- A separate classifier for spam, abuse, and generic fan comments.
- Better handling of replies, so reply chains are not treated as isolated comments.
- Better detection of copied comments and repeated slogans.
- Stronger multilingual handling (Clustering in other languages already works fairly well, but a lot of the cheap filters rely on hard-coded English keywords)

This stage matters because clustering only works well if the input comments are worth clustering.

### Better filtering after clustering

The current cluster quality step uses cheap rules before LLM scoring. It could be made stronger.

Possible additions:

- Penalise clusters where the representative comments do not agree with the keywords.
- Detect clusters that are mostly jokes, memes, insults, or meta-discussion.
- Split clusters that contain several unrelated arguments.
- Merge clusters that BERTopic separated but the LLM sees as the same argument.
- Use a second pass to produce a final ranked list of the most useful points.

### Better summaries

The LLM summary step could be improved by changing what is sent to the model.

Possible additions:

- Include more structured metadata about each comment.
- Send a better spread of comments from each cluster, not only the top representatives.
- Ask for separate fields: main claim, supporting reasons, criticism, evidence, suggested action, and disagreement.
- Ask the LLM to state when a cluster is too noisy to summarise.
- Add a final cross-cluster summary that removes duplication between clusters.

### Better UI

The current app shows cluster cards. Useful additions would be:

- Filters for highlighted, unscored, noisy, or low-confidence clusters.
- Sorting by usefulness, size, likes, replies, or noise.
- Export to Markdown, JSON, or CSV.
- A view that shows only the best comments across all clusters.
- A way to mark bad clusters and use that feedback to tune thresholds.

### Performance work

The slowest part is usually LLM scoring. There is a strong tradeoff between performance and quality. That balance is currently already tilting a bit too much towards prioritizing performance.

### Small tuning changes

Some useful small changes:

- Tune `CHORUS_PRECLUSTER_KEEP_FRACTION` per comment volume.
- Tune `CHORUS_LLM_CLUSTER_BUDGET` for speed versus summary coverage.
- Lower `CHORUS_CLUSTER_COMMENT_CHARS` if LLM calls are too slow.
- Raise `CHORUS_PRECLUSTER_MIN_SCORE` for noisy comment sections.
- Keep more comments before clustering when comment sections are small.
- Add tests with comment threads that are multilingual, toxic, repetitive, or very large.

## License

This project is licensed under the MIT License. See `LICENSE` for details.