Orthogonal and Stiefel constraints help control weight geometry, improve conditioning, and stabilize signal propagation in neural networks. However, standard Stiefel optimization fixes the represented singular values at one, preventing the model from learning direction-dependent amplification or attenuation. We introduce ManifoldFlow, a spectrum-learnable extension of Stiefel optimization that separates an orthonormal factor from a bounded symmetric positive-definite (SPD) spectral factor. The SPD factor equals the Gram matrix of the realized weight, providing exact spectral interpretability and recovering standard Stiefel optimization when set to the identity. ManifoldFlow uses the gradient component rejected by the Stiefel constraint to guide spectral adaptation. By aligning and accumulating this information over time, the method jointly updates the orthonormal basis and the bounded spectrum. We establish the geometry of the rejected gradient, characterize optimization on the Stiefel-SPD product space, and analyze how the learned spectrum affects conditioning, signal propagation, and spectral complexity. Across diverse tasks, ManifoldFlow consistently outperforms matched fixed-spectrum Stiefel baselines, with particularly strong gains in recurrent language modeling.

翻译：暂无翻译