Recent empirical studies have identified fixed point iteration phenomena in deep neural networks, where the hidden state tends to stabilize after several layers, showing minimal change in subsequent layers. This observation has spurred the development of practical methodologies, such as accelerating inference by bypassing certain layers once the hidden state stabilizes, selectively fine-tuning layers to modify the iteration process, and implementing loops of specific layers to maintain fixed point iterations. Despite these advancements, the understanding of fixed point iterations remains superficial, particularly in high-dimensional spaces, due to the inadequacy of current analytical tools. In this study, we conduct a detailed analysis of fixed point iterations in a vector-valued function modeled by neural networks. We establish a sufficient condition for the existence of multiple fixed points of looped neural networks based on varying input regions. Additionally, we expand our examination to include a robust version of fixed point iterations. To demonstrate the effectiveness and insights provided by our approach, we provide case studies that looped neural networks may exist $2^d$ number of robust fixed points under exponentiation or polynomial activation functions, where $d$ is the feature dimension. Furthermore, our preliminary empirical results support our theoretical findings. Our methodology enriches the toolkit available for analyzing fixed point iterations of deep neural networks and may enhance our comprehension of neural network mechanisms.

翻译：近期实证研究发现深度神经网络中存在定点迭代现象，即隐藏状态在若干层后趋于稳定，在后续层中变化极小。这一观察推动了实用方法的发展，例如在隐藏状态稳定后通过跳过某些层来加速推理、选择性微调层以修改迭代过程，以及实现特定层的循环以维持定点迭代。尽管取得了这些进展，由于当前分析工具的不足，对定点迭代的理解仍停留在表面，尤其是在高维空间中。本研究对神经网络建模的向量值函数中的定点迭代进行了详细分析。基于不同的输入区域，我们建立了循环神经网络存在多个定点的一个充分条件。此外，我们将分析扩展到包含定点迭代的鲁棒版本。为了展示我们方法的有效性和所提供的见解，我们提供了案例研究，表明在指数或多项式激活函数下，循环神经网络可能存在 $2^d$ 个鲁棒定点，其中 $d$ 是特征维度。此外，我们的初步实证结果支持了我们的理论发现。我们的方法丰富了分析深度神经网络定点迭代的可用工具集，并可能增进我们对神经网络机制的理解。