Test-time adaptation (TTA) for large language models (LLMs) updates model parameters at inference time using signals available at deployment. This paper focuses on a common yet under-explored regime: unsupervised, sample-specific TTA, where the model adapts independently for each prompt using only the prompt itself, without gold answers or external supervision. Although appealing, naive unsupervised TTA with a fixed, handcrafted learning rate can be unstable: updates may overfit to prompt-specific statistics, drift from the desired answer distribution, and ultimately degrade generation quality. This failure mode is not surprising, as in this case TTA must adapt to a single prompt within only a few gradient steps, unlike standard training that averages updates over large datasets and long optimization horizons. Therefore, we propose layer-wise dynamic test-time adaptation, a framework which explicitly modulates TTA strength as a function of prompt representation, LLM structure and adaptation step. In our setting, TTA updates only LoRA parameters, and a lightweight hypernetwork predicts per-layer, per-step learning-rate multipliers, enabling fine-grained control. Experiments across various datasets and LLMs consistently show that our method substantially strengthens TTA by learning effective scaling patterns over adaptation steps and transformer layer projections, improving stability while delivering better performance.

翻译：大语言模型（LLM）的测试时适应（TTA）利用部署时可用的信号在推理时更新模型参数。本文聚焦于一种常见但尚未充分探索的机制：无监督、样本特定的TTA，即模型仅依据提示本身独立地对每个提示进行适应，无需标准答案或外部监督。尽管这种方法颇具吸引力，但采用固定、人工设定学习率的朴素无监督TTA可能不稳定：更新可能过度拟合提示特定的统计特征，偏离期望的答案分布，最终降低生成质量。这种失效模式并不意外，因为在此情况下，TTA必须在仅有的几个梯度步内适应单个提示，这与基于大规模数据集和长优化周期进行更新平均的标准训练截然不同。因此，我们提出了分层动态测试时适应框架，该框架将TTA强度显式地调制为提示表示、LLM结构和适应步数的函数。在我们的设定中，TTA仅更新LoRA参数，并由一个轻量级超网络预测每层、每步的学习率乘子，从而实现细粒度控制。跨多种数据集和LLM的实验一致表明，我们的方法通过学习适应步数和Transformer层投影上的有效缩放模式，显著增强了TTA的稳定性，同时实现了更优的性能。