---
title: LLM Brainstorming
emoji: 🚀
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 5.34.2
python_version: 3.13
app_file: main.py
pinned: false
fullWidth: true
license: apache-2.0
short_description: Generate multiple answer and score filters to find the best
---

# llm-brainstorm

This project provides a small interface for running "tournaments" between language model answers. It is built with Gradio and LiteLLM.

## Usage

1. Create a `.env` file in the repository root and define any API keys required by your model. You can also set defaults for:
   - `NUM_TOP_PICKS`
   - `POOL_SIZE`
   - `MAX_WORKERS`
   - `NUM_GENERATIONS`
   - `OPENAI_API_BASE`
   - `OPENAI_API_KEY`
   - `GENERATE_MODEL`
   - `SCORE_MODEL`
   - `PAIRWISE_MODEL`
   - `GENERATE_TEMPERATURE`
   - `SCORE_TEMPERATURE`
   - `PAIRWISE_TEMPERATURE`
   - `PASS_INSTRUCTION_TO_SCORE`
   - `PASS_INSTRUCTION_TO_PAIRWISE`
   - `ENABLE_SCORE_FILTER`
   - `ENABLE_PAIRWISE_FILTER`
   - `ENABLE_GENERATE_THINKING`
   - `ENABLE_SCORE_THINKING`
   - `ENABLE_PAIRWISE_THINKING`

   When any of the thinking flags are enabled, the app sends
   `chat_template_kwargs={"enable_thinking": True}` with each
   `litellm.completion` call for that model. Otherwise it sends
   `chat_template_kwargs={"enable_thinking": False}`.

   The app uses [LiteLLM](https://github.com/BerriAI/litellm) to talk to
   language models. If you leave `OPENAI_API_BASE` blank, LiteLLM defaults to
   `https://api.openai.com/v1`. When the "API Token" field in the interface is
   empty, the value from `OPENAI_API_KEY` will be used. These defaults let you
   quickly connect to OpenAI without extra configuration.
2. Install dependencies (example with `pip`):
   ```bash
   pip install gradio litellm python-dotenv tqdm matplotlib
   ```
3. Run the app:
   ```bash
   python main.py
   ```
4. Open the displayed local URL. At the top of the page you can optionally override the API base path and token (the token field is blank by default). Additional settings let you configure score and pairwise filtering.

The interface will generate multiple answers, optionally filter them by score and run a pairwise tournament to select the best outputs. Results from previous pairwise comparisons are cached, so duplicate matches are skipped for faster tournaments. Pairwise results are aggregated using an Elo rating system to rank the players.

## Terminology

- *Judge* refers to both the **Score Model** and **Pairwise Model**.
- When instructions mention **updating the LLM**, update the **Generation**, **Score**, and **Pairwise** models together.
- *Output*, *answer*, or *response* are all considered the same as a **player** in the tournament.