The past decade has witnessed remarkable advancements in deep learning, owing to the emergence of various architectures, layers, objectives, and optimization techniques. These consist of a multitude of variations of attention, normalization, skip connections, transformer, and self-supervised learning methods, among others. Our goal is to furnish a comprehensive survey of significant recent contributions in these domains to individuals with a fundamental grasp of deep learning. Our aspiration is that an integrated and comprehensive approach of influential recent works will facilitate the formation of new connections between different areas of deep learning. In our discussion, we discuss multiple patterns that summarize the key strategies for many of the successful innovations over the last decade. We also include a discussion on recent commercially built, closed-source models such as OpenAI's GPT-4 and Google's PaLM 2.

翻译：过去十年间，得益于各类架构、层结构、目标函数及优化技术的涌现，深度学习领域取得了显著进展。这些进展包括注意力机制、归一化方法、跳跃连接、Transformer架构以及自监督学习方法等众多变体的发展。本文旨在为具备深度学习基础知识的读者提供该领域重要新成果的全面综述。我们期望通过整合与综合近期具有影响力的研究成果，能够促进深度学习不同领域之间形成新的关联。在讨论中，我们归纳了概括过去十年诸多成功创新核心策略的若干模式。此外，我们还探讨了近期商业化的闭源模型，例如OpenAI的GPT-4和Google的PaLM 2。