The metadata service (MDS) sits on the critical path for distributed file system (DFS) operations, and therefore it is key to the overall performance of a large-scale DFS. Common "serverful" MDS architectures, such as a single server or cluster of servers, have a significant shortcoming: either they are not scalable, or they make it difficult to achieve an optimal balance of performance, resource utilization, and cost. A modern MDS requires a novel architecture that addresses this shortcoming. To this end, we design and implement $\lambda$FS, an elastic, high-performance metadata service for large-scale DFSes. $\lambda$FS scales a DFS metadata cache elastically on a FaaS (Function-as-a-Service) platform and synthesizes a series of techniques to overcome the obstacles that are encountered when building large, stateful, and performance-sensitive applications on FaaS platforms. $\lambda$FS takes full advantage of the unique benefits offered by FaaS $\unicode{x2013}$ elastic scaling and massive parallelism $\unicode{x2013}$ to realize a highly-optimized metadata service capable of sustaining up to 4.13$\times$ higher throughput, 90.40% lower latency, 85.99% lower cost, 3.33$\times$ better performance-per-cost, and better resource utilization and efficiency than a state-of-the-art DFS for an industrial workload.
翻译:元数据服务(MDS)位于分布式文件系统(DFS)操作的关键路径上,因此它对大规模DFS的整体性能至关重要。常见的"有服务器"MDS架构(如单服务器或服务器集群)存在显著缺陷:它们要么无法扩展,要么难以在性能、资源利用率和成本之间实现最优平衡。现代MDS需要一种新型架构来解决这一缺陷。为此,我们设计并实现了$λ$FS——一种用于大规模DFS的弹性高性能元数据服务。$λ$FS在FaaS(函数即服务)平台上弹性扩展DFS元数据缓存,并综合运用一系列技术来克服在FaaS平台上构建大型、有状态且对性能敏感的应用程序时遇到的障碍。$λ$FS充分利用FaaS提供的独特优势——弹性扩展和大规模并行性——实现了一种高度优化的元数据服务,与工业负载下最先进的DFS相比,其吞吐量可提升至4.13倍,延迟降低90.40%,成本降低85.99%,性能成本比提升3.33倍,同时具有更优的资源利用率和效率。