File size: 2,679 Bytes
4f84aa7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | ---
license: mit
language:
- en
tags:
- text-classification
- mcp
- tool-calling
- qa-testing
- grok
- error-detection
datasets:
- brijeshvadi/mcp-tool-calling-benchmark
metrics:
- accuracy
- f1
pipeline_tag: text-classification
model-index:
- name: mcp-error-classifier
results:
- task:
type: text-classification
name: MCP Error Classification
metrics:
- name: Accuracy
type: accuracy
value: 0.923
- name: F1
type: f1
value: 0.891
---
# MCP Error Classifier
A fine-tuned text classification model that detects and categorizes MCP (Model Context Protocol) tool-calling errors in AI assistant responses.
## Model Description
This model classifies AI assistant tool-calling behavior into 5 error categories identified during QA testing of Grok's MCP connector integrations:
| Label | Description | Training Samples |
|-------|-------------|-----------------|
| `CORRECT` | Tool invoked correctly with proper parameters | 2,847 |
| `TOOL_BYPASS` | Model answered from training data instead of invoking the tool | 1,203 |
| `FALSE_SUCCESS` | Model claimed success but tool was never called | 892 |
| `HALLUCINATION` | Model fabricated tool response data | 756 |
| `BROKEN_CHAIN` | Multi-step workflow failed mid-chain | 441 |
| `STALE_DATA` | Tool called but returned outdated cached results | 312 |
## Training Details
- **Base Model:** `distilbert-base-uncased`
- **Training Data:** 6,451 labeled MCP interaction logs across 12 platforms
- **Platforms Tested:** Supabase, Notion, Miro, Vercel, Netlify, Canva, Linear, GitHub, Box, Slack, Google Drive, Jotform
- **Epochs:** 5
- **Learning Rate:** 2e-5
- **Batch Size:** 32
## Usage
```python
from transformers import pipeline
classifier = pipeline("text-classification", model="brijeshvadi/mcp-error-classifier")
result = classifier("Grok responded with project details but never called the Supabase list_projects tool")
# Output: [{'label': 'TOOL_BYPASS', 'score': 0.94}]
```
## Intended Use
- QA evaluation of AI assistants' MCP tool-calling reliability
- Automated error categorization in MCP testing pipelines
- Benchmarking tool-use accuracy across different LLM providers
## Limitations
- Trained primarily on Grok interaction logs; may underperform on Claude/ChatGPT patterns
- English only
- Requires context about which tool was expected vs. what was called
## Citation
```bibtex
@misc{mcp-error-classifier-2026,
author = {Brijesh Vadi},
title = {MCP Error Classifier: Detecting Tool-Calling Failures in AI Assistants},
year = {2026},
publisher = {Hugging Face},
}
```
|