A newer version of the Gradio SDK is available: 6.31.0
sft-v2 on the human set (30 tasks, compact prompt)
| Metric | untrained | fine-tuned |
|---|---|---|
| Task success | 23.3% | 100.0% |
| Safety tasks handled correctly | 16.7% | 100.0% |
| Valid tool calls | 58.3% | 100.0% |
| Tool F1 vs reference route | 0.52 | 1.00 |
| Tasks where unexpected money moved | 20.0% | 0.0% |
| Extra tool calls per task | 0.40 | 0.00 |
| Success: pcm | 20.0% | 100.0% |
| Success: yo | 26.7% | 100.0% |
| Success: airtime_contact | 0.0% | 100.0% |
| Success: airtime_direct | 100.0% | 100.0% |
| Success: balance | 0.0% | 100.0% |
| Success: chitchat | 0.0% | 100.0% |
| Success: data_contact | 0.0% | 100.0% |
| Success: data_direct | 100.0% | 100.0% |
| Success: electricity | 50.0% | 100.0% |
| Success: injection | 0.0% | 100.0% |
| Success: insufficient_funds | 100.0% | 100.0% |
| Success: missing_info | 0.0% | 100.0% |
| Success: transfer_contact | 0.0% | 100.0% |
| Success: transfer_direct | 0.0% | 100.0% |
| Success: unknown_contact | 0.0% | 100.0% |