Achieving efficient and robust multi-channel data learning is a challenging task in data science. By exploiting low-rankness in the transformed domain, i.e., transformed low-rankness, tensor Singular Value Decomposition (t-SVD) has achieved extensive success in multi-channel data representation and has recently been extended to function representation such as Neural Networks with t-product layers (t-NNs). However, it still remains unclear how t-SVD theoretically affects the learning behavior of t-NNs. This paper is the first to answer this question by deriving the upper bounds of the generalization error of both standard and adversarially trained t-NNs. It reveals that the t-NNs compressed by exact transformed low-rank parameterization can achieve a sharper adversarial generalization bound. In practice, although t-NNs rarely have exactly transformed low-rank weights, our analysis further shows that by adversarial training with gradient flow (GF), the over-parameterized t-NNs with ReLU activations are trained with implicit regularization towards transformed low-rank parameterization under certain conditions. We also establish adversarial generalization bounds for t-NNs with approximately transformed low-rank weights. Our analysis indicates that the transformed low-rank parameterization can promisingly enhance robust generalization for t-NNs.

翻译：实现高效且鲁棒的多通道数据学习是数据科学中的一项挑战性任务。通过利用变换域中的低秩性（即变换低秩性），张量奇异值分解（t-SVD）已在多通道数据表示中取得广泛成功，并近期被扩展至函数表示领域，例如具有t-乘积层的神经网络（t-NNs）。然而，t-SVD如何从理论上影响t-NNs的学习行为仍不明确。本文首次通过推导标准与对抗训练t-NNs的泛化误差上界回答了该问题。研究表明，采用精确变换低秩参数化压缩的t-NNs可获得更优的对抗泛化界。实际中，尽管t-NNs的权重极少具有精确的变换低秩性，我们的进一步分析表明，在梯度流（GF）对抗训练下，过参数化的ReLU激活t-NNs会在特定条件下通过隐式正则化趋向于变换低秩参数化。我们还为近似变换低秩权重的t-NNs建立了对抗泛化界。分析表明，变换低秩参数化有望显著增强t-NNs的鲁棒泛化能力。