Compositional priors describe the generic properties of layered functions in deep Bayesian models, where deep neural networks with random weights are a canonical example.In the wide-network limit, the prior is a Gaussian process with a depth-dependent kernel, and its behaviour as depth grows has been extensively studied through this kernel. Here, we study another case, where each layer itself is a vector valued Gaussian process, and our aim is similarly to understand the limiting behaviour of the prior as depth grows. Previous GP work has established that for the RBF kernel and a certain range of bandwidths $r$, the prior degenerates in the limit, converging to the set of constant functions -- which is not useful as a probabilistic model. In this paper we establish several new results. First, we identify a sharp bandwidth threshold $r_c(d) = Θ(\sqrt{d})$ above which the limit is degenerate, strengthening the earlier bounds. Second, and more importantly, we show that for $r$ below the threshold $r_c(d)$ the prior converges to a limit distribution $π_{\bar{Z}}$. We also prove that these distributions are non-degenerate and non-Gaussian, with non-vanishing dependence between coordinates. In contrast to the previously known degenerate regime, deep Gaussian process priors can therefore admit non-trivial limits. Empirically, we verify the threshold across a range of dimensions $d$, and demonstrate a complex multimodal behaviour of the limit distributions $π_{\bar{Z}}$ -- a regime that becomes increasingly narrow with $d$ and would be hard to identify without knowing the threshold.

翻译：组合先验描述了深度贝叶斯模型中分层函数的通用性质，其中随机权重的深度神经网络是典型范例。在宽网络极限下，先验是具有深度相关核的高斯过程，其随深度增长的行为已通过该核被广泛研究。本文研究另一种情形：每层本身是向量值高斯过程，我们的目标同样是理解先验随深度增长时的极限行为。既有高斯过程研究表明：对于RBF核及特定带宽范围$r$，该先验在极限下退化，收敛至常数函数集合——这作为概率模型毫无价值。本文建立了若干新结论。首先，我们识别出尖锐的带宽阈值$r_c(d) = Θ(\sqrt{d})$：当带宽高于该阈值时，极限退化，从而强化了先前的界。其次，更重要的是，我们证明当$r$低于阈值$r_c(d)$时，先验收敛至极限分布$π_{\bar{Z}}$。我们还证明这些分布非退化且非高斯，坐标间存在非零依赖关系。与先前已知的退化机制不同，深度高斯过程先验因此可接纳非平凡极限。实验上，我们验证了跨维度$d$的阈值，并展示了极限分布$π_{\bar{Z}}$复杂的多模态行为——该机制随$d$增加而急剧收窄，且若不确知阈值则难以识别。