Effective research ideation is a critical step for scientific research. However, the exponential increase in scientific literature makes it challenging for researchers to stay current with recent advances and identify meaningful research directions. Recent developments in large language models~(LLMs) suggest a promising avenue for automating the generation of novel research ideas. However, existing methods for idea generation either trivially prompt LLMs or directly expose LLMs to extensive literature without indicating useful information. Inspired by the research process of human researchers, we propose a Chain-of-Ideas~(CoI) agent, an LLM-based agent that organizes relevant literature in a chain structure to effectively mirror the progressive development in a research domain. This organization facilitates LLMs to capture the current advancements in research, thereby enhancing their ideation capabilities. Furthermore, we propose Idea Arena, an evaluation protocol that can comprehensively evaluate idea generation methods from different perspectives, aligning closely with the preferences of human researchers. Experimental results indicate that the CoI agent consistently outperforms other methods and shows comparable quality as humans in research idea generation. Moreover, our CoI agent is budget-friendly, with a minimum cost of \$0.50 to generate a candidate idea and its corresponding experimental design.

翻译：有效的研究构思是科学研究的关键步骤。然而，科学文献的指数级增长使得研究人员难以及时跟进最新进展并识别有意义的研究方向。大型语言模型（LLMs）的最新发展为自动化生成新颖研究思想提供了有前景的途径。然而，现有的思想生成方法要么简单地提示LLMs，要么直接将LLMs暴露于大量文献而未指明有效信息。受人类研究者研究过程的启发，我们提出思想链（CoI）智能体——一种基于LLM的智能体，它以链式结构组织相关文献，有效映射研究领域的渐进发展。这种组织方式有助于LLMs捕捉当前研究进展，从而增强其构思能力。此外，我们提出思想竞技场评估协议，该协议能够从不同角度全面评估思想生成方法，并与人类研究者的偏好紧密契合。实验结果表明，CoI智能体始终优于其他方法，在研究思想生成方面展现出与人类相当的质量。此外，我们的CoI智能体具有成本效益，生成一个候选思想及其对应实验设计的最低成本仅为0.50美元。