Agentic AI systems that invoke external tools are powerful but costly, leading developers to default to large models and overspend inference budgets. Existing model routers target chat responses or end-task success, not the correctness of the individual tool calls on which agent tasks depend; we present Switchcraft, the first (to the best of our knowledge) model router optimized for strict tool-call correctness in agentic workloads. We construct an evaluation framework on six function-calling benchmarks, holding one out entirely, and fine-tune a calibrated DistilBERT-based classifier, deployed inline under a latency budget, to select the lowest-cost model subject to correctness or abstain when none is likely to answer correctly. Switchcraft reaches 78.3% answer accuracy (86.4% when correct abstentions are credited) at 59% lower cost than the best individual model (75.6%), saving over $3,800 per million queries, and outperforms kNN, MIRT, heuristic, and random routers. On the held-out benchmark, it matches or exceeds every individual model at 5--7X lower cost than the strongest ones. We find two counter-intuitive results: larger models do not consistently outperform smaller ones on tool-use tasks, and nominally cheaper models can incur higher total cost due to token-intensive output.

翻译：暂无翻译