|
|
<!DOCTYPE html> |
|
|
<html lang="en"> |
|
|
<head> |
|
|
<meta charset="UTF-8"> |
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0"> |
|
|
<title>LLM Tool-Calling Models: Cost & Context Analysis</title> |
|
|
<link rel="stylesheet" href="style.css"> |
|
|
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js"></script> |
|
|
<script src="https://cdn.jsdelivr.net/npm/chartjs-plugin-annotation@3.0.1/dist/chartjs-plugin-annotation.min.js"></script> |
|
|
<script src="https://cdn.jsdelivr.net/npm/papaparse@5.4.1/papaparse.min.js"></script> |
|
|
</head> |
|
|
<body> |
|
|
<header> |
|
|
<h1>Agentic LLM Price Comparison</h1> |
|
|
<p class="subtitle">Cost-effectiveness analysis of 218 tool-calling models from OpenRouter</p> |
|
|
<p class="date">Data snapshot: November 8, 2025</p> |
|
|
<nav class="header-nav"> |
|
|
<a href="output-input-analysis.html">🔢 Output/Input Price Analysis →</a> |
|
|
</nav> |
|
|
</header> |
|
|
|
|
|
<main> |
|
|
<section class="intro"> |
|
|
<h2>Overview</h2> |
|
|
<p>This analysis examines LLM models with tool-calling capabilities - essential for agentic AI systems, MCP (Model Context Protocol) implementations, and multi-tool orchestration. Models are evaluated across two key dimensions:</p> |
|
|
<ul> |
|
|
<li><strong>Cost per token</strong> - Economic efficiency for production use</li> |
|
|
<li><strong>Context window size</strong> - Capability for complex, long-form tasks</li> |
|
|
</ul> |
|
|
|
|
|
<div class="attribution"> |
|
|
<h3>Methodology</h3> |
|
|
<p><strong>Quadrant Classification:</strong> Models are categorized into four quadrants based on median values calculated from the dataset:</p> |
|
|
<ul> |
|
|
<li><strong>Cost Division:</strong> Median average cost (mean of input + output pricing) = <span id="median-cost-display">calculated from data</span></li> |
|
|
<li><strong>Context Division:</strong> Median context window size = <span id="median-context-display">calculated from data</span></li> |
|
|
</ul> |
|
|
<p>This creates four categories: Low Cost/High Context, High Cost/High Context, Low Cost/Low Context, and High Cost/Low Context.</p> |
|
|
|
|
|
<h3>Attribution & Data Notes</h3> |
|
|
<ul> |
|
|
<li><strong>Data Collection:</strong> <a href="https://danielrosehill.com" target="_blank">Daniel Rosehill</a></li> |
|
|
<li><strong>Collection Date:</strong> November 8, 2025</li> |
|
|
<li><strong>Source:</strong> Prices derived from API calls to <a href="https://openrouter.ai/" target="_blank">OpenRouter</a></li> |
|
|
<li><strong>Pricing Variability:</strong> OpenRouter pricing can fluctuate slightly according to the end inference provider. The same models may have slightly different pricing even at the time of capture.</li> |
|
|
<li><strong>Data Exclusions:</strong> Free models (cost = $0) were excluded from this analysis</li> |
|
|
</ul> |
|
|
</div> |
|
|
</section> |
|
|
|
|
|
<section class="stats-grid"> |
|
|
<div class="stat-card"> |
|
|
<div class="stat-number" id="total-models">218</div> |
|
|
<div class="stat-label">Models Analyzed</div> |
|
|
</div> |
|
|
<div class="stat-card"> |
|
|
<div class="stat-number" id="vendors">-</div> |
|
|
<div class="stat-label">Vendors</div> |
|
|
</div> |
|
|
<div class="stat-card"> |
|
|
<div class="stat-number" id="median-input">-</div> |
|
|
<div class="stat-label">Median Input ($/M)</div> |
|
|
</div> |
|
|
<div class="stat-card"> |
|
|
<div class="stat-number" id="median-output">-</div> |
|
|
<div class="stat-label">Median Output ($/M)</div> |
|
|
</div> |
|
|
<div class="stat-card"> |
|
|
<div class="stat-number" id="median-multiple">-</div> |
|
|
<div class="stat-label">Median Out/In Multiple</div> |
|
|
</div> |
|
|
<div class="stat-card"> |
|
|
<div class="stat-number" id="context-range">-</div> |
|
|
<div class="stat-label">Context Range</div> |
|
|
</div> |
|
|
</section> |
|
|
|
|
|
<div class="tabs"> |
|
|
<button class="tab-button active" data-tab="models">All Models</button> |
|
|
<button class="tab-button" data-tab="quadrants">Quadrant Analysis</button> |
|
|
<button class="tab-button" data-tab="analysis">Visual Analysis</button> |
|
|
</div> |
|
|
|
|
|
<div class="tab-content active" id="tab-models"> |
|
|
<section class="data-notes"> |
|
|
<h3>Data Notes</h3> |
|
|
<ul> |
|
|
<li><strong>Cost Calculation:</strong> Input and output pricing shown separately (per million tokens)</li> |
|
|
<li><strong>Data Exclusions:</strong> Free models (cost = $0) were excluded from this analysis</li> |
|
|
<li><strong>Color Coding:</strong> Cells are color-coded by tier - see legend below for ranges</li> |
|
|
</ul> |
|
|
</section> |
|
|
|
|
|
<section class="data-table-section"> |
|
|
<h2>All Models - Comprehensive View</h2> |
|
|
<p>Complete dataset with color-coded cost and context tiers. Click column headers to sort.</p> |
|
|
|
|
|
<div class="tier-legends"> |
|
|
<div class="legend-group"> |
|
|
<h4>Cost Tiers ($/M tokens)</h4> |
|
|
<div class="legend-items"> |
|
|
<div class="legend-item"><span class="tier-indicator cost-very-low"></span> Very Low (< $0.10)</div> |
|
|
<div class="legend-item"><span class="tier-indicator cost-low"></span> Low ($0.10 - $0.50)</div> |
|
|
<div class="legend-item"><span class="tier-indicator cost-medium"></span> Medium ($0.50 - $2.00)</div> |
|
|
<div class="legend-item"><span class="tier-indicator cost-high"></span> High ($2.00 - $5.00)</div> |
|
|
<div class="legend-item"><span class="tier-indicator cost-very-high"></span> Very High (> $5.00)</div> |
|
|
</div> |
|
|
</div> |
|
|
<div class="legend-group"> |
|
|
<h4>Context Window Tiers</h4> |
|
|
<div class="legend-items"> |
|
|
<div class="legend-item"><span class="tier-indicator context-small"></span> Small (< 32K)</div> |
|
|
<div class="legend-item"><span class="tier-indicator context-medium"></span> Medium (32K - 128K)</div> |
|
|
<div class="legend-item"><span class="tier-indicator context-large"></span> Large (128K - 256K)</div> |
|
|
<div class="legend-item"><span class="tier-indicator context-very-large"></span> Very Large (256K - 1M)</div> |
|
|
<div class="legend-item"><span class="tier-indicator context-ultra"></span> Ultra (> 1M)</div> |
|
|
</div> |
|
|
</div> |
|
|
</div> |
|
|
<div class="table-controls"> |
|
|
<input type="text" id="search" placeholder="Search models..." /> |
|
|
<select id="quadrant-filter"> |
|
|
<option value="">All Quadrants</option> |
|
|
<option value="Low Cost / High Context">Low Cost / High Context</option> |
|
|
<option value="High Cost / High Context">High Cost / High Context</option> |
|
|
<option value="Low Cost / Low Context">Low Cost / Low Context</option> |
|
|
<option value="High Cost / Low Context">High Cost / Low Context</option> |
|
|
</select> |
|
|
</div> |
|
|
<div class="table-wrapper"> |
|
|
<table id="models-table"> |
|
|
<thead> |
|
|
<tr> |
|
|
<th class="sortable" data-sort="model_name">Model Name <span class="sort-indicator"></span></th> |
|
|
<th class="sortable" data-sort="displayVendor">Vendor <span class="sort-indicator"></span></th> |
|
|
<th class="sortable" data-sort="context_length">Context <span class="sort-indicator"></span></th> |
|
|
<th class="sortable" data-sort="input_price_usd_per_m">Input ($/M) <span class="sort-indicator"></span></th> |
|
|
<th class="sortable" data-sort="output_price_usd_per_m">Output ($/M) <span class="sort-indicator"></span></th> |
|
|
<th class="sortable" data-sort="output_input_multiple">Out/In Multiple <span class="sort-indicator"></span></th> |
|
|
<th class="sortable" data-sort="quadrant">Quadrant <span class="sort-indicator"></span></th> |
|
|
</tr> |
|
|
</thead> |
|
|
<tbody id="table-body"> |
|
|
|
|
|
</tbody> |
|
|
</table> |
|
|
</div> |
|
|
</section> |
|
|
</div> |
|
|
|
|
|
<div class="tab-content" id="tab-quadrants"> |
|
|
<section class="data-notes"> |
|
|
<h3>Quadrant Analysis</h3> |
|
|
<ul> |
|
|
<li><strong>Overview:</strong> Models divided into four quadrants based on median cost and context values</li> |
|
|
<li><strong>Green (Low Cost / High Context):</strong> Best value - low prices with large context windows</li> |
|
|
<li><strong>Blue (High Cost / High Context):</strong> Premium models with extensive context capabilities</li> |
|
|
<li><strong>Orange (Low Cost / Low Context):</strong> Budget-friendly for simple tasks</li> |
|
|
<li><strong>Red (High Cost / Low Context):</strong> Specialized models with limited context</li> |
|
|
</ul> |
|
|
</section> |
|
|
|
|
|
<section class="chart-section"> |
|
|
<h2>All Quadrants Overview</h2> |
|
|
<p>Complete view showing all models across quadrants with median division lines.</p> |
|
|
<div class="chart-container"> |
|
|
<canvas id="allQuadrantsChart"></canvas> |
|
|
</div> |
|
|
</section> |
|
|
|
|
|
<section class="quadrant-details"> |
|
|
<h2>Individual Quadrant Deep Dive</h2> |
|
|
<p>Detailed view of each quadrant's models for easier comparison within each category.</p> |
|
|
|
|
|
<div class="quadrant-grid"> |
|
|
<div class="quadrant-detail"> |
|
|
<h3 style="color: #10b981;">Low Cost / High Context</h3> |
|
|
<div class="chart-container-small"> |
|
|
<canvas id="quadrant-lchc"></canvas> |
|
|
</div> |
|
|
</div> |
|
|
|
|
|
<div class="quadrant-detail"> |
|
|
<h3 style="color: #2563eb;">High Cost / High Context</h3> |
|
|
<div class="chart-container-small"> |
|
|
<canvas id="quadrant-hchc"></canvas> |
|
|
</div> |
|
|
</div> |
|
|
|
|
|
<div class="quadrant-detail"> |
|
|
<h3 style="color: #f59e0b;">Low Cost / Low Context</h3> |
|
|
<div class="chart-container-small"> |
|
|
<canvas id="quadrant-lclc"></canvas> |
|
|
</div> |
|
|
</div> |
|
|
|
|
|
<div class="quadrant-detail"> |
|
|
<h3 style="color: #ef4444;">High Cost / Low Context</h3> |
|
|
<div class="chart-container-small"> |
|
|
<canvas id="quadrant-hclc"></canvas> |
|
|
</div> |
|
|
</div> |
|
|
</div> |
|
|
</section> |
|
|
</div> |
|
|
|
|
|
<div class="tab-content" id="tab-analysis"> |
|
|
<section class="data-notes"> |
|
|
<h3>Analysis Notes</h3> |
|
|
<ul> |
|
|
<li><strong>Quadrant Divisions:</strong> Based on median values for cost and context window size</li> |
|
|
<li><strong>Logarithmic Scale:</strong> Cost axis uses log scale to better visualize the wide price range</li> |
|
|
<li><strong>Median Lines:</strong> Dashed lines show median cost and context values across all models</li> |
|
|
</ul> |
|
|
</section> |
|
|
|
|
|
<section class="chart-section"> |
|
|
<h2>Cost vs Context Quadrant Analysis</h2> |
|
|
<p>Interactive scatter plot showing the relationship between pricing and context window size. Hover over points for detailed model information.</p> |
|
|
<div class="chart-container"> |
|
|
<canvas id="quadrantChart"></canvas> |
|
|
</div> |
|
|
</section> |
|
|
</div> |
|
|
</main> |
|
|
|
|
|
<footer> |
|
|
<p>Data source: <a href="https://openrouter.ai/" target="_blank">OpenRouter API</a> | Analysis: <a href="https://github.com/danielrosehill/OR-Models-With-Tools-0811" target="_blank">GitHub Repository</a></p> |
|
|
<p class="disclaimer">Prices are subject to change. For current pricing, always refer to OpenRouter's live API.</p> |
|
|
</footer> |
|
|
|
|
|
<script src="app.js"></script> |
|
|
</body> |
|
|
</html> |
|
|
|
