# Token Probability Analyzer

A web application that analyzes token probabilities using various language models. This tool helps visualize and understand how language models predict tokens in a given text sequence.

## Features

- Support for multiple language models (GPT-2, TinyLlama, etc.)
- Token-by-token probability analysis
- Percentile scoring for token probabilities
- Top-k predictions for each position
- Joint and average log likelihood calculations

## Setup

1. Install the required dependencies:
```bash
pip install -r requirements.txt
```

2. Run the application:
```bash
python app.py
```

3. Open your browser and navigate to `http://localhost:5000`

## Usage

1. Select a language model from the dropdown menu
2. Enter your text in the input field
3. Click "Analyze" to see the token probabilities and predictions

## Technical Details

The application uses Flask for the backend and provides a simple web interface. It leverages the Hugging Face Transformers library to load and run various language models for token probability analysis.