Label	Description	Training Samples
`CORRECT`	Tool invoked correctly with proper parameters	2,847
`TOOL_BYPASS`	Model answered from training data instead of invoking the tool	1,203
`FALSE_SUCCESS`	Model claimed success but tool was never called	892
`HALLUCINATION`	Model fabricated tool response data	756
`BROKEN_CHAIN`	Multi-step workflow failed mid-chain	441
`STALE_DATA`	Tool called but returned outdated cached results	312

Training Details

Base Model: distilbert-base-uncased
Training Data: 6,451 labeled MCP interaction logs across 12 platforms
Platforms Tested: Supabase, Notion, Miro, Vercel, Netlify, Canva, Linear, GitHub, Box, Slack, Google Drive, Jotform
Epochs: 5
Learning Rate: 2e-5
Batch Size: 32

Usage

python

from transformers import pipeline

classifier = pipeline("text-classification", model="brijeshvadi/mcp-error-classifier")

result = classifier("Grok responded with project details but never called the Supabase list_projects tool")
# Output: [{'label': 'TOOL_BYPASS', 'score': 0.94}]

Intended Use

QA evaluation of AI assistants' MCP tool-calling reliability
Automated error categorization in MCP testing pipelines
Benchmarking tool-use accuracy across different LLM providers

Limitations

Trained primarily on Grok interaction logs; may underperform on Claude/ChatGPT patterns
English only
Requires context about which tool was expected vs. what was called

Citation

bibtex

@misc{mcp-error-classifier-2026,
  author = {Brijesh Vadi},
  title = {MCP Error Classifier: Detecting Tool-Calling Failures in AI Assistants},
  year = {2026},
  publisher = {Hugging Face},
}