Local LLM Tool Use Benchmark for TrashClaw
TrashClaw operates heavily on autonomous tool execution (file editing, shell commands, semantic search). Not all local models handle these complex instructions equally.
Below is a benchmark of popular local LLMs tested for their capability to act as the cognitive engine for TrashClaw, focusing on function calling reliability, XML fallback parsing, multi-step orchestration, and edit boundary respect.
Model Compatibility Matrix
| Model | Native Tool Calling | XML Fallback Parsing | Multi-step Tasks (3+ tools) | Respects File Edit Boundaries | Overall Grade | Notes |
|---|---|---|---|---|---|---|
| Qwen 2.5 Coder (32B) | 🟢 Excellent | 🟢 Excellent | 🟢 Excellent | 🟢 Excellent | S | The current gold standard for local tool use. Flawlessly orchestrates multi-step workflows and handles complex file editing without hallucinating boundaries. |
| Qwen 2.5 Coder (14B) | 🟢 Good | 🟢 Excellent | 🟡 Fair | 🟢 Good | A- | Very capable for standard operations. Occasionally struggles to chain more than 3 tools in a single context window without losing track of the final goal. |
| Qwen 2.5 Coder (7B) | 🟡 Fair | 🟢 Good | 🔴 Poor | 🟡 Fair | B- | Good for single-shot tasks (e.g., "read this file"). Tends to loop or forget parameters during complex multi-step orchestration. |
| Llama 3.1 (8B Instruct) | 🔴 Poor | 🟢 Good | 🔴 Poor | 🟡 Fair | C | Native tool calling format often breaks or hallucinates schemas. However, it performs decently when forced into the XML fallback parsing mode. |
| Llama 3.3 (70B Instruct) | 🟢 Excellent | 🟢 Excellent | 🟢 Good | 🟢 Excellent | A | Highly capable and reliable, though slower than Qwen 32B on similar hardware. Great at respecting system prompts and edit boundaries. |
| DeepSeek Coder V2 (16B) | 🟢 Good | 🟢 Good | 🟢 Good | 🟢 Excellent | A- | Excellent at the actual code editing (replacements/diffs), but can sometimes be stubborn about using tools iteratively instead of trying to write bash scripts to do everything at once. |
| DeepSeek Coder V2 (Lite) | 🟡 Fair | 🟡 Fair | 🔴 Poor | 🟡 Fair | C+ | Often hallucinates tool names or provides malformed JSON arguments when native tool calling is enabled. |
| Mistral Nemo (12B) | 🟢 Good | 🟢 Good | 🟡 Fair | 🟡 Fair | B+ | Solid generalist. Handles basic tool schemas well but struggles with the strict character-matching required for replace_string operations. |
| Mixtral 8x7B Instruct | 🟡 Fair | 🟢 Good | 🟡 Fair | 🔴 Poor | C+ | Tends to ignore tool instructions in favor of just printing the code to the user. Requires very strong system prompt constraints. |
| Phi-3 Mini (3.8B) | 🔴 Poor | 🟡 Fair | 🔴 Poor | 🔴 Poor | D | Too small for reliable agentic workflows. Fails to adhere to JSON schemas and frequently truncates outputs. |
Evaluation Criteria
- Native Tool Calling: How reliably does the model format its output according to the OpenAI/Ollama tool calling JSON schema without syntax errors?
- XML Fallback Parsing: If native tools fail, can the model reliably fall back to using
<tool_call>XML tags as instructed in the system prompt? - Multi-step Tasks: Can the model autonomously execute a sequence of actions (e.g.,
search_files->read_file->edit_file) to solve an ambiguous goal without human intervention? - Respects File Edit Boundaries: When using editing tools (like
replace_string), does the model provide exact, unescaped literal text, or does it lazily use// ... rest of codeplaceholders which break the patching mechanism?
Recommendations for TrashClaw
- Best Performance (32GB+ RAM):
qwen2.5-coder:32borllama3.3 - Best Balance (16GB RAM):
deepseek-coder-v2orqwen2.5-coder:14b - Best for Low-end Hardware (8GB RAM):
qwen2.5-coder:7b(Ensure you use XML fallback mode for better reliability).