Recent advances in autonomous LLM agents demonstrate their ability to improve performance through iterative interaction with the environment. We define this paradigm as Test-Time Improvement (TTI). However, the mechanisms under how and why TTI succeed or fail remain poorly understood, and existing evaluation metrics fail to capture their task optimization efficiency, behavior adaptation after erroneous actions, and the specific utility of working memory for task completion. To address these gaps, we propose Test-time Improvement Diagnostic Evaluation (TIDE), an agent-agnostic and environment-agnostic framework that decomposes TTI into three comprehensive and interconnected dimensions. The framework measures (1) the overall temporal dynamics of task completion and (2) identifies whether performance is primarily constrained by recursive looping behaviors or (3) by burdensome accumulated memory. Through extensive experiments across diverse agents and environments, TIDE highlights that improving agent performance requires more than scaling internal reasoning, calling for explicitly optimizing the interaction dynamics between the agent and the environment.

翻译：近年来，自主LLM智能体通过与环境进行迭代交互以提升性能的能力得到了显著发展。我们将这一范式定义为测试时改进。然而，TTI为何成功或失败的内在机制仍不甚明晰，且现有评估指标未能充分捕捉其任务优化效率、错误行动后的行为适应性，以及工作记忆对任务完成的具体效用。为填补这些空白，我们提出了测试时改进诊断评估框架，这是一个与智能体及环境无关的框架，将TTI分解为三个相互关联的综合性维度。该框架度量（1）任务完成的整体时序动态，并（2）识别性能受限的主要原因是递归循环行为，还是（3）沉重的累积记忆负担。通过在多样化智能体与环境中的大量实验，TIDE表明，提升智能体性能不仅需要扩展内部推理能力，更需显式优化智能体与环境间的交互动态。