We consider online model selection with decentralized data over $M$ clients, and study a fundamental problem: the necessity of collaboration. Previous work gave a negative answer from the perspective of worst-case regret minimization, while we give a different answer from the perspective of regret-computational cost trade-off. We separately propose a federated algorithm with and without communication constraint and prove regret bounds that show (i) collaboration is unnecessary if we do not limit the computational cost on each client; (ii) collaboration is necessary if we limit the computational cost on each client to $o(K)$, where $K$ is the number of candidate hypothesis spaces. As a by-product, we improve the regret bounds of algorithms for distributed online multi-kernel learning at a smaller computational and communication cost. Our algorithms rely on three new techniques, i.e., an improved Bernstein's inequality for martingale, a federated algorithmic framework, named FOMD-No-LU, and decoupling model selection and predictions, which might be of independent interest.

翻译：我们考虑在 $M$ 个客户端上使用去中心化数据进行在线模型选择，并研究一个基本问题：协作的必要性。以往的工作从最坏情况遗憾最小化的角度给出了否定答案，而我们则从遗憾-计算成本权衡的角度给出了不同答案。我们分别提出了无通信约束和有通信约束的联邦算法，并证明了遗憾上界，表明：（i）如果我们不限制每个客户端的计算成本，则协作是不必要的；（ii）如果我们限制每个客户端的计算成本为 $o(K)$（其中 $K$ 是候选假设空间的数量），则协作是必要的。作为副产品，我们以更小的计算和通信成本改进了分布式在线多核学习算法的遗憾上界。我们的算法依赖于三项新技术，即改进的鞅 Bernstein 不等式、名为 FOMD-No-LU 的联邦算法框架，以及解耦模型选择与预测，这些技术可能具有独立的研究价值。