We study the problem of parameter-free stochastic optimization, inquiring whether, and under what conditions, do fully parameter-free methods exist: these are methods that achieve convergence rates competitive with optimally tuned methods, without requiring significant knowledge of the true problem parameters. Existing parameter-free methods can only be considered ``partially'' parameter-free, as they require some non-trivial knowledge of the true problem parameters, such as a bound on the stochastic gradient norms, a bound on the distance to a minimizer, etc. In the non-convex setting, we demonstrate that a simple hyperparameter search technique results in a fully parameter-free method that outperforms more sophisticated state-of-the-art algorithms. We also provide a similar result in the convex setting with access to noisy function values under mild noise assumptions. Finally, assuming only access to stochastic gradients, we establish a lower bound that renders fully parameter-free stochastic convex optimization infeasible, and provide a method which is (partially) parameter-free up to the limit indicated by our lower bound.

翻译：我们研究参数自由的随机优化问题，探究在何种条件下才能实现完全参数自由的方法：这些方法能够在无需了解真实问题参数关键信息的情况下，达到与经过最优调节的方法相媲美的收敛速度。现有参数自由方法只能被视为“部分”参数自由，因为它们仍需要某些非平凡的真实问题参数知识，例如随机梯度范数的界限、到最小点的距离界限等。在非凸优化场景中，我们证明一种简单的超参数搜索技术能够构建出完全参数自由的方法，其性能优于更复杂的先进算法。在温和噪声假设下，我们针对可获取含噪函数值的凸优化场景，也给出了类似结果。最后，在仅能访问随机梯度的条件下，我们建立了下界，证明完全参数自由的随机凸优化不可行，并给出了一种达到该下界所指示极限的（部分）参数自由方法。