Large language models (LLMs) have made rapid advancements in code generation for popular languages such as Python and C++. Many of these recent gains can be attributed to the use of ``agents'' that wrap domain-relevant tools alongside LLMs. Hardware design languages such as Verilog have also seen improved code generation in recent years, but the impact of agentic frameworks on Verilog code generation tasks remains unclear. In this work, we present the first systematic evaluation of agentic LLMs for Verilog generation, using the recently introduced CVDP benchmark. We also introduce several open-source hardware design agent harnesses, providing a model-agnostic baseline for future work. Through controlled experiments across frontier models, we study how structured prompting and tool design affect performance, analyze agent failure modes and tool usage patterns, compare open-source and closed-source models, and provide qualitative examples of successful and failed agent runs. Our results show that naive agentic wrapping around frontier models can degrade performance (relative to standard forward passes with optimized prompts), but that structured harnesses meaningfully match and in some cases exceed non-agentic baselines. We find that the performance gap between open and closed source models is driven by both higher crash rates and weaker tool output interpretation. Our exploration illuminates the path towards designing special-purpose agents for verilog generation in the future.

翻译：大型语言模型（LLMs）在Python和C++等流行语言的代码生成领域取得了快速进展。这些近期成果中有许多可归因于使用“智能体”——即围绕领域相关工具包装LLM的系统。硬件描述语言（如Verilog）近年来在代码生成方面也有所改进，但智能体框架对Verilog代码生成任务的影响尚不明确。在本工作中，我们首次针对Verilog生成任务开展了智能体LLM的系统性评估，采用了近期提出的CVDP基准测试。同时，我们引入多个开源硬件设计智能体框架，为未来研究提供模型无关的基线。通过在前沿模型间开展受控实验，我们研究了结构化提示和工具设计如何影响性能，分析了智能体的失败模式与工具使用模式，对比了开源与闭源模型，并提供了成功与失败智能体运行的定性案例。结果表明：在前沿模型上简单包装智能体会导致性能下降（相较于采用优化提示的标准前向传播），但结构化框架能有意义地匹配甚至在某些情况下超越非智能体基线。我们发现，开源与闭源模型的性能差距既源于更高的崩溃率，也源于更弱的工具输出解析能力。本探索为未来设计面向Verilog生成的专用智能体指明了路径。