Client-side metadata caching has long been considered an effective method for accelerating metadata operations in distributed file systems (DFSs). However, we have found that client-side state (e.g., caching) is not only ineffective but also consumes valuable memory resources in the deep learning pipelines. We thus propose FalconFS, a DFS optimized for deep learning pipelines with the stateless-client architecture. Specifically, instead of performing client-side path resolution and caching, FalconFS efficiently resolves paths on the server side using hybrid metadata indexing and lazy namespace replication. FalconFS also boosts server concurrency with concurrent request merging and provides easy deployment with VFS shortcut. Evaluations against CephFS and Lustre show that FalconFS achieves up to 5.72$\times$ throughput for small file read/write and up to 12.81$\times$ throughput for deep learning model training. FalconFS has been running in Huawei autonomous driving system's production environment with 10,000 NPUs for one year and has been open-sourced.

翻译：客户端元数据缓存长期以来被认为是加速分布式文件系统（DFS）中元数据操作的有效方法。然而，我们发现客户端状态（例如缓存）在深度学习流水线中不仅效果不佳，还会消耗宝贵的内存资源。因此，我们提出了FalconFS，一种采用无状态客户端架构、专为深度学习流水线优化的DFS。具体而言，FalconFS不在客户端执行路径解析和缓存，而是通过混合元数据索引和惰性命名空间复制，在服务器端高效地解析路径。FalconFS还通过并发请求合并提升服务器并发能力，并通过VFS快捷方式实现便捷部署。与CephFS和Lustre的对比评估表明，FalconFS在小文件读写上实现了高达5.72倍的吞吐量，在深度学习模型训练上实现了高达12.81倍的吞吐量。FalconFS已在华为自动驾驶系统的生产环境中运行一年，部署规模达10,000个NPU，并已开源。