SGD and AdamW are the two most used optimizers for fine-tuning large neural networks in computer vision. When the two methods perform the same, SGD is preferable because it uses less memory (12 bytes/parameter with momentum and 8 bytes/parameter without) than AdamW (16 bytes/parameter). However, on a suite of downstream tasks, especially those with distribution shifts, we find that fine-tuning with AdamW performs substantially better than SGD on modern Vision Transformer and ConvNeXt models. We find that large gaps in performance between SGD and AdamW occur when the fine-tuning gradients in the first "embedding" layer are much larger than in the rest of the model. Our analysis suggests an easy fix that works consistently across datasets and models: freezing the embedding layer (less than 1% of the parameters) leads to SGD with or without momentum performing slightly better than AdamW while using less memory (e.g., on ViT-L, SGD uses 33% less GPU memory). Our insights result in state-of-the-art accuracies on five popular distribution shift benchmarks: WILDS-FMoW, WILDS-Camelyon, BREEDS-Living-17, Waterbirds, and DomainNet.

翻译：SGD和AdamW是计算机视觉中微调大型神经网络最常用的两种优化器。当两种方法性能相同时，SGD更优，因其内存占用（带动量时为12字节/参数，不带动量时为8字节/参数）低于AdamW（16字节/参数）。然而，在一系列下游任务中（特别是存在分布偏移的任务），我们发现使用AdamW微调现代Vision Transformer和ConvNeXt模型的性能显著优于SGD。我们注意到，当第一个"嵌入"层的微调梯度远大于模型其他层时，SGD与AdamW之间会出现较大的性能差距。我们的分析提出了一种简单且跨数据集、模型一致的改进方案：冻结嵌入层（参数占比不足1%）可使带或不带动量的SGD略优于AdamW，同时内存占用更低（例如，在ViT-L上，SGD的GPU内存使用减少33%）。这一发现使我们在五个主流分布偏移基准测试中取得了最先进的精度：WILDS-FMoW、WILDS-Camelyon、BREEDS-Living-17、Waterbirds和DomainNet。