Modern table formats such as Apache Iceberg compute and store metadata-commit timestamps, record counts, and column-level statistics such as null counts and value bounds at write time as part of file writing. These statistics serve query planning, yet they overlap substantially with data quality (DQ) monitoring needs. We describe a metadata-first approach that repurposes write-time statistics for continuous DQ observability: anomaly detection, drift monitoring, null-rate tracking; without scanning any data. Deployed at LinkedIn across 200,000+ Iceberg tables (800+ PB), this approach satisfies approximately 60% of user-defined DQ rules at zero marginal compute cost and reduces profiling resource consumption by around 50%. Extending manifest statistics with lightweight counters (sum, zero-value counts, boolean counts) and incrementally mergeable sketches; Theta sketches for distinct counts, KLL sketches for quantiles; can further raise metadata-satisfiable coverage to close to 90% of production DQ rules. We validate sketch accuracy, mergeability, and storage overhead on production data and propose that table formats should store per-file sketches in Puffin sidecar files, following the same store-then-aggregate pattern used for existing manifest statistics.

翻译：现代表格式（如Apache Iceberg）在写入文件时即计算并存储元数据提交时间戳、记录数以及列级统计信息（如空值计数和值范围）。这些统计信息服务于查询规划，但同时也与数据质量监测需求高度重叠。我们提出一种元数据优先的方法，将写入时统计信息重新用于持续的数据质量可观测性：异常检测、漂移监控、空值率追踪，全程无需扫描任何数据。该方法已在LinkedIn部署于20万+Iceberg表（800+PB），以零边际计算成本满足约60%的用户定义数据质量规则，并将画像资源消耗降低约50%。通过扩展清单统计信息（增加轻量计数器：总和、零值计数、布尔计数）以及增量可合并草图（用于去重计数的Theta草图、用于分位数的KLL草图），可将元数据可满足规则覆盖率提升至生产环境数据质量规则的近90%。我们在生产数据上验证了草图的准确性、可合并性及存储开销，并提出表格式应将每个文件的草图存储在Puffin辅助文件中，遵循与现有清单统计信息相同的“存储-聚合”模式。