Recurrent Neural Cascades (RNCs) are the recurrent neural networks with no cyclic dependencies among recurrent neurons. This class of recurrent networks has received a lot of attention in practice. Besides training methods for a fixed architecture such as backpropagation, the cascade architecture naturally allows for constructive learning methods, where recurrent nodes are added incrementally one at a time, often yielding smaller networks. Furthermore, acyclicity amounts to a structural prior that even for the same number of neurons yields a more favourable sample complexity compared to a fully-connected architecture. A central question is whether the advantages of the cascade architecture come at the cost of a reduced expressivity. We provide new insights into this question. We show that the regular languages captured by RNCs with sign and tanh activation with positive recurrent weights are the star-free regular languages. In order to establish our results we developed a novel framework where capabilities of RNCs are accessed by analysing which semigroups and groups a single neuron is able to implement. A notable implication of our framework is that RNCs can achieve the expressivity of all regular languages by introducing neurons that can implement groups.

翻译：递归神经级联（RNCs）是指循环神经元之间不存在循环依赖关系的递归神经网络。这类循环网络在实践中受到了广泛关注。除了针对固定架构的训练方法（如反向传播）外，级联架构自然支持增量式构造学习方法，即每次逐步添加一个循环节点，通常能生成更小的网络。此外，非循环性构成了一种结构先验，即使神经元数量相同，其样本复杂度也优于全连接架构。核心问题在于：级联架构的优势是否以降低表达能力为代价？我们为这一问题提供了新见解。研究表明，采用sign和tanh激活函数且具有正循环权重的RNCs所能捕获的正则语言是星自由正则语言。为建立这一结果，我们开发了一个新颖框架，通过分析单个神经元能实现的半群和群来评估RNCs的能力。该框架的一个重要推论是：通过引入能实现群的神经元，RNCs可达到所有正则语言的表达能力。