Neural networks offer good approximation to many tasks but consistently fail to reach perfect generalization, even when theoretical work shows that such perfect solutions can be expressed by certain architectures. Using the task of formal language learning, we focus on one simple formal language and show that the theoretically correct solution is in fact not an optimum of commonly used objectives -- even with regularization techniques that according to common wisdom should lead to simple weights and good generalization (L1, L2) or other meta-heuristics (early-stopping, dropout). However, replacing standard targets with the Minimum Description Length objective (MDL) results in the correct solution being an optimum.

翻译：神经网络在众多任务中能提供良好的近似，但始终未能达到完美泛化，即使理论研究已证明此类完美解可由特定架构表达。以形式语言学习任务为例，我们聚焦于一种简单形式语言，揭示出理论上的正确解实际上并非常用目标的优化目标——即便采用根据常识应能产生简单权重并实现良好泛化的正则化技术（L1、L2正则化）或其他元启发式方法（早停法、丢弃法）。然而，将标准目标替换为最小描述长度目标后，正确解将成为优化目标。