Large-scale Earth system datasets, from high-resolution remote sensing imagery to spatiotemporal climate model outputs, exhibit characteristics analogous to those of standard videos. Their inherent spatial, temporal, and spectral redundancies can thus be readily exploited by established video compression techniques. Here, we present xarrayvideo, a Python library for compressing multichannel spatiotemporal datasets by encoding them as videos. Our approach achieves compression ratios of up to 250x while maintaining high fidelity by leveraging standard, well-optimized video codecs through ffmpeg. We demonstrate the library's effectiveness on four real-world multichannel spatiotemporal datasets: DynamicEarthNet (very high resolution Planet images), DeepExtremeCubes (high resolution Sentinel-2 images), ERA5 (weather reanalysis data), and the SimpleS2 dataset (high resolution multichannel Sentinel-2 images), achieving Peak Signal-to-Noise Ratios (PSNRs) of 55.86, 40.60, 46.58, and 43.23 dB at 0.1 bits per pixel per band (bpppb) and 65.91, 54.28, 62.90, and 55.04 dB at 1 bpppb. We are redistributing two of these datasets, DeepExtremeCubes (2.3 Tb) and DynamicEarthNet (525 Gb), in the machine-learning-ready and cloud-ready TACO format through HuggingFace at significantly reduced sizes (270 Gb and 8.5 Gb, respectively) without compromising quality (PSNR 55.77-56.65 and 60.15). No performance loss is observed when the compressed versions of these datasets are used in their respective deep learning-based downstream tasks (next step reflectance prediction and landcover segmentation). In conclusion, xarrayvideo presents an efficient solution for handling the rapidly growing size of Earth observation datasets, making advanced compression techniques accessible and practical to the Earth science community. The library is available for use at https://github.com/IPL-UV/xarrayvideo

翻译：大规模地球系统数据集，从高分辨率遥感影像到时空气候模型输出，展现出与标准视频相似的特征。因此，其固有的空间、时间和光谱冗余性可以利用成熟的视频压缩技术进行有效利用。本文介绍了xarrayvideo，一个用于通过将多通道时空数据集编码为视频进行压缩的Python库。我们的方法通过ffmpeg利用标准且经过充分优化的视频编解码器，在保持高保真度的同时实现了高达250倍的压缩比。我们在四个真实世界的多通道时空数据集上验证了该库的有效性：DynamicEarthNet（超高分辨率Planet影像）、DeepExtremeCubes（高分辨率Sentinel-2影像）、ERA5（天气再分析数据）以及SimpleS2数据集（高分辨率多通道Sentinel-2影像），在每像素每波段0.1比特（bpppb）下实现了55.86、40.60、46.58和43.23 dB的峰值信噪比（PSNR），在1 bpppb下实现了65.91、54.28、62.90和55.04 dB的PSNR。我们通过HuggingFace以显著减小的体积（分别为270 Gb和8.5 Gb）重新发布了其中两个数据集——DeepExtremeCubes（2.3 Tb）和DynamicEarthNet（525 Gb），采用机器学习就绪且云就绪的TACO格式，且未牺牲质量（PSNR分别为55.77-56.65 dB和60.15 dB）。当这些数据集的压缩版本用于其各自基于深度学习的下游任务（下一步反射率预测和土地覆盖分割）时，未观察到性能损失。总之，xarrayvideo为处理快速增长的地球观测数据规模提供了一个高效的解决方案，使先进的压缩技术对地球科学界而言易于获取且实用。该库可在 https://github.com/IPL-UV/xarrayvideo 获取使用。