Large language models (LLMs) have made rapid advancements in code generation for popular languages such as Python and C++. Many of these recent gains can be attributed to the use of ``agents'' that wrap domain-relevant tools alongside LLMs. Hardware design languages such as Verilog have also seen improved code generation in recent years, but the impact of agentic frameworks on Verilog code generation tasks remains unclear. In this work, we present the first systematic evaluation of agentic LLMs for Verilog generation, using the recently introduced CVDP benchmark. We also introduce several open-source hardware design agent harnesses, providing a model-agnostic baseline for future work. Through controlled experiments across frontier models, we study how structured prompting and tool design affect performance, analyze agent failure modes and tool usage patterns, compare open-source and closed-source models, and provide qualitative examples of successful and failed agent runs. Our results show that naive agentic wrapping around frontier models can degrade performance (relative to standard forward passes with optimized prompts), but that structured harnesses meaningfully match and in some cases exceed non-agentic baselines. We find that the performance gap between open and closed source models is driven by both higher crash rates and weaker tool output interpretation. Our exploration illuminates the path towards designing special-purpose agents for verilog generation in the future.

翻译：大型语言模型（LLMs）在Python和C++等流行语言的代码生成方面取得了快速进展。这些近期成果很大程度上归因于使用了将领域相关工具与LLMs相结合的“智能体”。硬件设计语言如Verilog在近年来也出现了代码生成能力的提升，但智能体框架对Verilog代码生成任务的影响仍不明确。在这项工作中，我们利用近期推出的CVDP基准，首次对用于Verilog生成的智能体LLMs进行了系统性评估。我们还引入了多个开源硬件设计智能体工具集，为未来工作提供了与模型无关的基线。通过在前沿模型上进行的受控实验，我们研究了结构化提示与工具设计如何影响性能，分析了智能体故障模式与工具使用模式，比较了开源与闭源模型，并提供了成功与失败智能体运行过程的定性示例。我们的结果表明，围绕前沿模型进行简单的智能体封装会降低性能（相对于使用优化提示的标准前向传播），但结构化的工具集能显著匹配甚至在某些情况下超越非智能体基线。我们发现，开源与闭源模型之间的性能差距既源于更高的崩溃率，也源于较弱的工具输出解读能力。我们的探索为未来设计Verilog生成的专用智能体指明了路径。