Progressing beyond centralized AI is of paramount importance, yet, distributed AI solutions, in particular various federated learning (FL) algorithms, are often not comprehensively assessed, which prevents the research community from identifying the most promising approaches and practitioners from being convinced that a certain solution is deployment-ready. The largest hurdle towards FL algorithm evaluation is the difficulty of conducting real-world experiments over a variety of FL client devices and different platforms, with different datasets and data distribution, all while assessing various dimensions of algorithm performance, such as inference accuracy, energy consumption, and time to convergence, to name a few. In this paper, we present CoLExT, a real-world testbed for FL research. CoLExT is designed to streamline experimentation with custom FL algorithms in a rich testbed configuration space, with a large number of heterogeneous edge devices, ranging from single-board computers to smartphones, and provides real-time collection and visualization of a variety of metrics through automatic instrumentation. According to our evaluation, porting FL algorithms to CoLExT requires minimal involvement from the developer, and the instrumentation introduces minimal resource usage overhead. Furthermore, through an initial investigation involving popular FL algorithms running on CoLExT, we reveal previously unknown trade-offs, inefficiencies, and programming bugs.

翻译：超越集中式人工智能至关重要，然而分布式人工智能解决方案，特别是各类联邦学习算法，往往未能得到全面评估，这阻碍了研究界识别最有前景的方法，也使从业者难以确信特定解决方案已具备部署条件。联邦学习算法评估的最大障碍在于难以在多样化联邦学习客户端设备与不同平台上开展真实世界实验，这些实验需涵盖不同数据集与数据分布，同时评估算法性能的多个维度，例如推理准确率、能耗及收敛时间等。本文提出CoLExT——一个面向联邦学习研究的真实世界测试平台。CoLExT旨在通过丰富的测试配置空间简化自定义联邦学习算法的实验流程，该平台配备大量异构边缘设备（从单板计算机到智能手机），并通过自动化检测工具实现多种指标的实时收集与可视化。评估表明，将联邦学习算法移植至CoLExT仅需开发者极少量介入，且检测机制引入的资源开销微乎其微。此外，通过在CoLExT上运行主流联邦学习算法的初步研究发现，我们揭示了先前未知的权衡关系、效率缺陷及程序错误。