Large GPU clusters for AI research rely on POSIX-based storage spanning code authoring, data preparation, and model checkpointing. The metric that matters is researcher iteration speed, not just peak throughput - but current benchmarks target the latter: synthetic tools stress peak bandwidth and IOPS, emulators replay captured training I/O, and neither execute real framework operations or captures the bursty, heterogeneous, metadata-heavy patterns of research. We present PRISM, an extensible benchmark suite that executes real workloads across key stages of AI research including version control, environment setup, data preparation, data loading, model checkpointing, and synthetic data generation. PRISM exercises what researchers actually run and thereby serves not only as a benchmark but also as a qualification gate for new storage systems before they are consumed by researchers. We share our experience using PRISM for over 18 months in AI research clusters with thousands of Nvidia hopper class GPUs consuming petabyte scale Lustre and NFS based storage systems in characterizing research workloads, identifying the appropriate system for different cluster environments and detecting performance regressions. As an example, PRISM surfaced an openat lock contention pathology that caused an 8x slowdown under Fully Sharded Data Parallel (FSDP) checkpointing which went uncaught in emulated benchmarks. We plan to open source PRISM as a practical tool for selecting and validating storage for AI research infrastructure.

翻译：暂无翻译