| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - text-classification |
| - mcp |
| - tool-calling |
| - qa-testing |
| - grok |
| - error-detection |
| datasets: |
| - brijeshvadi/mcp-tool-calling-benchmark |
| metrics: |
| - accuracy |
| - f1 |
| pipeline_tag: text-classification |
| model-index: |
| - name: mcp-error-classifier |
| results: |
| - task: |
| type: text-classification |
| name: MCP Error Classification |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 0.923 |
| - name: F1 |
| type: f1 |
| value: 0.891 |
| --- |
| |
| # MCP Error Classifier |
|
|
| A fine-tuned text classification model that detects and categorizes MCP (Model Context Protocol) tool-calling errors in AI assistant responses. |
|
|
| ## Model Description |
|
|
| This model classifies AI assistant tool-calling behavior into 5 error categories identified during QA testing of Grok's MCP connector integrations: |
|
|
| | Label | Description | Training Samples | |
| |-------|-------------|-----------------| |
| | `CORRECT` | Tool invoked correctly with proper parameters | 2,847 | |
| | `TOOL_BYPASS` | Model answered from training data instead of invoking the tool | 1,203 | |
| | `FALSE_SUCCESS` | Model claimed success but tool was never called | 892 | |
| | `HALLUCINATION` | Model fabricated tool response data | 756 | |
| | `BROKEN_CHAIN` | Multi-step workflow failed mid-chain | 441 | |
| | `STALE_DATA` | Tool called but returned outdated cached results | 312 | |
|
|
| ## Training Details |
|
|
| - **Base Model:** `distilbert-base-uncased` |
| - **Training Data:** 6,451 labeled MCP interaction logs across 12 platforms |
| - **Platforms Tested:** Supabase, Notion, Miro, Vercel, Netlify, Canva, Linear, GitHub, Box, Slack, Google Drive, Jotform |
| - **Epochs:** 5 |
| - **Learning Rate:** 2e-5 |
| - **Batch Size:** 32 |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import pipeline |
| |
| classifier = pipeline("text-classification", model="brijeshvadi/mcp-error-classifier") |
| |
| result = classifier("Grok responded with project details but never called the Supabase list_projects tool") |
| # Output: [{'label': 'TOOL_BYPASS', 'score': 0.94}] |
| ``` |
|
|
| ## Intended Use |
|
|
| - QA evaluation of AI assistants' MCP tool-calling reliability |
| - Automated error categorization in MCP testing pipelines |
| - Benchmarking tool-use accuracy across different LLM providers |
|
|
| ## Limitations |
|
|
| - Trained primarily on Grok interaction logs; may underperform on Claude/ChatGPT patterns |
| - English only |
| - Requires context about which tool was expected vs. what was called |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{mcp-error-classifier-2026, |
| author = {Brijesh Vadi}, |
| title = {MCP Error Classifier: Detecting Tool-Calling Failures in AI Assistants}, |
| year = {2026}, |
| publisher = {Hugging Face}, |
| } |
| ``` |
|
|
