Large Language Model (LLM) image recognition is a powerful tool for extracting data from images, but accuracy depends on providing sufficient cues in the prompt - requiring a domain expert for specialized tasks. We introduce Cue Learning using Evolution for Accurate Recognition (CLEAR), which uses a combination of LLMs and evolutionary computation to generate and optimize cues such that recognition of specialized features in images is improved. It achieves this by auto-generating a novel domain-specific representation and then using it to optimize suitable textual cues with a genetic algorithm. We apply CLEAR to the real-world task of identifying sustainability data from interior and exterior images of buildings. We investigate the effects of using a variable-length representation compared to fixed-length and show how LLM consistency can be improved by refactoring from categorical to real-valued estimates. We show that CLEAR enables higher accuracy compared to expert human recognition and human-authored prompts in every task with error rates improved by up to two orders of magnitude and an ablation study evincing solution concision.

翻译：大型语言模型（LLM）图像识别是从图像中提取数据的强大工具，但其准确性取决于提示中是否提供充分线索——这在专业任务中需要领域专家参与。我们提出基于进化的线索学习用于精确识别（CLEAR），该方法结合LLM与进化计算来生成并优化线索，从而提升图像中专业特征的识别能力。其实现机制是：首先生成一种新颖的领域特定表征，进而利用遗传算法优化对应的文本线索。我们将CLEAR应用于从建筑内外图像识别可持续性数据的实际任务中，探究了变长表征相较于定长表征的效果，并演示如何通过将分类估计重构为实值估计来提升LLM的一致性。实验表明，在所有任务中CLEAR均优于专家人工识别及人工撰写的提示，错误率降低达两个数量级，消融实验亦验证了解决方案的简洁性。