Vision Language Models (VLMs) have rapidly advanced and show strong promise for text-based person search (TBPS), a task that requires capturing fine-grained relationships between images and text to distinguish individuals. Previous methods address these challenges through local alignment, yet they are often prone to shortcut learning and spurious correlations, yielding misalignment. Moreover, injecting prior knowledge can distort intra-modality structure. Motivated by our finding that encoder attention surfaces spatially precise evidence from the earliest training epochs, and to alleviate these issues, we introduceITSELF, an attention-guided framework for implicit local alignment. At its core, Guided Representation with Attentive Bank (GRAB) converts the model's own attention into an Attentive Bank of high-saliency tokens and applies local objectives on this bank, learning fine-grained correspondences without extra supervision. To make the selection reliable and non-redundant, we introduce Multi-Layer Attention for Robust Selection (MARS), which aggregates attention across layers and performs diversity-aware top-k selection; and Adaptive Token Scheduler (ATS), which schedules the retention budget from coarse to fine over training, preserving context early while progressively focusing on discriminative details. Extensive experiments on three widely used TBPS benchmarks showstate-of-the-art performance and strong cross-dataset generalization, confirming the effectiveness and robustness of our approach without additional prior supervision. Our project is publicly available at https://trhuuloc.github.io/itself

翻译：视觉语言模型（VLMs）的快速发展为基于文本的人物搜索（TBPS）任务展现出巨大潜力，该任务需要捕捉图像与文本间的细粒度关系以区分不同个体。现有方法通常通过局部对齐应对这些挑战，但往往容易陷入捷径学习与虚假关联，导致对齐偏差。此外，引入先验知识可能扭曲模态内部结构。基于我们发现编码器注意力在训练初期即可提取空间精确证据的特性，为缓解上述问题，本文提出ITSELF——一种注意力引导的隐式局部对齐框架。其核心组件“注意力库引导表征”将模型自身的注意力转化为高显著性标记构成的注意力库，并在此库上施加局部优化目标，从而在无额外监督的情况下学习细粒度对应关系。为确保选择的可靠性与非冗余性，我们提出“多层注意力鲁棒选择”机制，通过跨层注意力聚合与多样性感知的top-k选择实现稳定筛选；以及“自适应标记调度器”，在训练过程中由粗到细动态调整保留预算，早期保留上下文信息的同时逐步聚焦判别性细节。在三个广泛使用的TBPS基准测试上的大量实验表明，该方法取得了最先进的性能并展现出强大的跨数据集泛化能力，证实了本方法在无需额外先验监督情况下的有效性与鲁棒性。项目已开源：https://trhuuloc.github.io/itself