For software that relies on machine-learned functionality, model selection is key to finding the right model for the task with desired performance characteristics. Evaluating a model requires developers to i) select from many models (e.g. the Hugging face model repository), ii) select evaluation metrics and training strategy, and iii) tailor trade-offs based on the problem domain. However, current evaluation approaches are either ad-hoc resulting in sub-optimal model selection or brute force leading to wasted compute. In this work, we present \toolname, a novel tool to automatically select and evaluate models based on the application scenario provided in natural language. We leverage the reasoning capabilities of large language models to propose a training strategy and extract desired trade-offs from a problem description. \toolname~features a resource-efficient experimentation engine that integrates constraints and trade-offs based on the problem into the model selection process. Our preliminary evaluation demonstrates that \toolname{} is both efficient and accurate compared to ad-hoc evaluations and brute force. This work presents an important step toward energy-efficient tools to help reduce the environmental impact caused by the growing demand for software with machine-learned functionality.

翻译：对于依赖机器学习功能的软件而言，模型选择是找到符合任务需求且具备预期性能特征模型的关键。评估模型需要开发者：i) 从众多模型（如Hugging Face模型库）中进行选择，ii) 选定评估指标与训练策略，iii) 根据问题领域进行权衡取舍。然而，当前的评估方法要么导致次优模型选择的临时性做法，要么造成计算资源浪费的暴力搜索。本文提出一种名为\toolname的新型工具，可基于自然语言描述的应用场景自动选择并评估模型。我们利用大语言模型的推理能力提出训练策略，并从问题描述中提取所需的权衡因素。\toolname配备资源高效的实验引擎，能将基于问题的约束与权衡纳入模型选择过程。初步评估表明，与临时性评估和暴力搜索相比，\toolname在效率与准确性方面均表现优异。本工作为实现高能效工具迈出了重要一步，有助于减少因机器学习功能软件需求增长而带来的环境影响。