README.md

metadata

title: Token Probability Visualization API
emoji: 📊
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 4.44.0
app_file: app.py
pinned: false
short_description: API for visualizing LLM token probabilities and embeddings

Token Probability Visualization API

This Space provides API endpoints for visualizing token probabilities and embeddings from language models.

Features

• Token Probabilities: Get next-token probability distributions for any input text
• Embeddings: Access token embeddings with PCA projections for visualization
• Comparative Analysis: Compare conditional vs unconditional distributions
• Multiple Contexts: Analyze probability shifts across different text contexts

API Endpoints

/predict - Main Prediction Endpoint

Returns comprehensive token probabilities, embeddings, and PCA projections.

/embeddings - Token Embeddings

Get embeddings for specific token IDs.

Usage

Use the API to build interactive visualizations showing:

• Token probability landscapes
• Embedding space visualizations
• Probability distribution comparisons
• Token movement analysis

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference