Accurate in-season crop type classification is crucial for the crop production estimation and monitoring of agricultural parcels. However, the complexity of the plant growth patterns and their spatio-temporal variability present significant challenges. While current deep learning-based methods show promise in crop type classification from single- and multi-modal time series, most existing methods rely on a single modality, such as satellite optical remote sensing data or crop rotation patterns. We propose a novel approach to fuse multimodal information into a model for improved accuracy and robustness across multiple years and countries. The approach relies on three modalities used: remote sensing time series from Sentinel-2 and Landsat 8 observations, parcel crop rotation and local crop distribution. To evaluate our approach, we release a new annotated dataset of 7.4 million agricultural parcels in France and Netherlands. We associate each parcel with time-series of surface reflectance (Red and NIR) and biophysical variables (LAI, FAPAR). Additionally, we propose a new approach to automatically aggregate crop types into a hierarchical class structure for meaningful model evaluation and a novel data-augmentation technique for early-season classification. Performance of the multimodal approach was assessed at different aggregation level in the semantic domain spanning from 151 to 8 crop types or groups. It resulted in accuracy ranging from 91\% to 95\% for NL dataset and from 85\% to 89\% for FR dataset. Pre-training on a dataset improves domain adaptation between countries, allowing for cross-domain zero-shot learning, and robustness of the performances in a few-shot setting from France to Netherlands. Our proposed approach outperforms comparable methods by enabling learning methods to use the often overlooked spatio-temporal context of parcels, resulting in increased preci...

翻译：准确的当季作物类型分类对于作物产量估算及农业地块监测至关重要。然而，植物生长模式的复杂性及其时空变异性带来了显著挑战。尽管当前基于深度学习的单模态与多模态时间序列方法在作物类型分类中展现出前景，但多数现有方法仍依赖单一数据源，如卫星光学遥感数据或作物轮作模式。我们提出了一种融合多模态信息的新方法，以提升模型在多年份、多国家场景下的准确性与鲁棒性。该方法依托三种模态：Sentinel-2与Landsat 8观测的遥感时间序列、地块轮作模式及局部作物分布。为评估该方法，我们发布了包含法国与荷兰740万农业地块的新标注数据集，为每个地块关联了地表反射率（红波段与近红外波段）及生物物理变量（叶面积指数、光合有效辐射吸收比例）的时间序列。此外，我们提出了一种自动将作物类型聚合为层次化类别结构的新方法以实现有意义的模型评估，并设计了一种用于早期分类的新型数据增强技术。在从151种到8种作物类别或组的语义域不同聚合层级上评估了多模态方法的性能。结果表明，该方法在荷兰数据集上准确率达91%至95%，在法国数据集上准确率达85%至89%。基于数据集进行预训练可提升国家间的域适应能力，实现跨域零样本学习，并在从法国到荷兰的小样本场景中保持鲁棒性能。通过使学习方法能够利用常被忽视的地块时空上下文，我们提出的方法优于同类对比方法，从而显著提升预测精度……