Automated machine learning (AutoML) methods improve upon existing models by optimizing various aspects of their design. While present methods focus on hyperparameters and neural network topologies, other aspects of neural network design can be optimized as well. To further the state of the art in AutoML, this dissertation introduces techniques for discovering more powerful activation functions and establishing more robust weight initialization for neural networks. These contributions improve performance, but also provide new perspectives on neural network optimization. First, the dissertation demonstrates that discovering solutions specialized to specific architectures and tasks gives better performance than reusing general approaches. Second, it shows that jointly optimizing different components of neural networks is synergistic, and results in better performance than optimizing individual components alone. Third, it demonstrates that learned representations are easier to optimize than hard-coded ones, creating further opportunities for AutoML. The dissertation thus makes concrete progress towards fully automatic machine learning in the future.

翻译：自动化机器学习（AutoML）方法通过优化设计的多个方面来改进现有模型。虽然现有方法主要关注超参数和神经网络拓扑结构，但神经网络设计的其他方面同样可以优化。为推进AutoML技术的发展前沿，本论文引入了发现更具表达力的激活函数以及建立更鲁棒的神经网络权重初始化的技术。这些贡献不仅提升了性能，还为神经网络优化提供了新视角。首先，论文证明针对特定架构和任务发现专用解决方案比复用通用方法能带来更优性能。其次，研究表明联合优化神经网络的不同组件具有协同效应，且比单独优化各组件能获得更优性能。第三，论文证实学习得到的表征比硬编码表征更易优化，从而为AutoML创造更多可能性。综上，本论文为实现未来全自动机器学习取得了具体进展。