Retrieving relevant observations from long multi-modal web interaction histories is challenging because relevance depends on the evolving task state, modality (screenshots, HTML text, structured signals), and temporal distance. Prior approaches typically rely on static similarity thresholds or fixed-capacity buffers, which fail to adapt relevance to the current task context. We propose \textbf{ACGM}, a learned graph-memory retriever that constructs \emph{task-adaptive} relevance graphs over agent histories using policy-gradient optimization from downstream task success. ACGM captures heterogeneous temporal dynamics with modality-specific decay (visual decays $4.3\times$ faster than text: $λ_v{=}0.47$ vs.\ $λ_x{=}0.11$) and learns sparse connectivity (3.2 edges/node), enabling efficient $O(\log T)$ retrieval. Across WebShop, VisualWebArena, and Mind2Web, ACGM improves retrieval quality to \textbf{82.7 nDCG@10} (+9.3 over GPT-4o, $p{<}0.001$) and \textbf{89.2\% Precision@10} (+7.7), outperforming 19 strong dense, re-ranking, multi-modal, and graph-based baselines. Code to reproduce our results is available at{\color{blue}\href{https://github.com/S-Forouzandeh/ACGM-Agentic-Web}{Saman Forouzandeh}}.

翻译：从长期多模态网络交互历史中检索相关观测结果具有挑战性，因为相关性取决于演进中的任务状态、模态（截图、HTML文本、结构化信号）及时序距离。先前方法通常依赖静态相似度阈值或固定容量缓冲区，未能根据当前任务上下文调整相关性。我们提出\textbf{ACGM}，一种基于学习图记忆的检索器，通过下游任务成功的策略梯度优化，在智能体历史中构建\textit{任务自适应}相关性图。ACGM通过模态特异性衰减（视觉衰减速度比文本快4.3倍：$λ_v{=}0.47$ vs.\ $λ_x{=}0.11$）建模异质时序动态，并学习稀疏连接（3.2条边/节点），实现高效的$O(\log T)$检索。在WebShop、VisualWebArena和Mind2Web上，ACGM将检索质量提升至\textbf{82.7 nDCG@10}（较GPT-4o提升9.3，$p{<}0.001$）和\textbf{89.2\% Precision@10}（提升7.7），超越了19种强基线方法，包括稠密检索、重排序、多模态及图模型。完整可复现代码已发布于{\color{blue}\href{https://github.com/S-Forouzandeh/ACGM-Agentic-Web}{Saman Forouzandeh}}。