Nutrition estimation is crucial for effective dietary management and overall health and well-being. Existing methods often struggle with sub-optimal accuracy and can be time-consuming. In this paper, we propose NuNet, a transformer-based network designed for nutrition estimation that utilizes both RGB and depth information from food images. We have designed and implemented a multi-scale encoder and decoder, along with two types of feature fusion modules, specialized for estimating five nutritional factors. These modules effectively balance the efficiency and effectiveness of feature extraction with flexible usage of our customized attention mechanisms and fusion strategies. Our experimental study shows that NuNet outperforms its variants and existing solutions significantly for nutrition estimation. It achieves an error rate of 15.65%, the lowest known to us, largely due to our multi-scale architecture and fusion modules. This research holds practical values for dietary management with huge potential for transnational research and deployment and could inspire other applications involving multiple data types with varying degrees of importance.

翻译：营养估算对于有效的膳食管理及整体健康福祉至关重要。现有方法常因精度欠佳且耗时较长而面临挑战。本文提出NuNet，一种基于Transformer架构的网络，专门用于营养估算，其同时利用食物图像的RGB信息与深度信息。我们设计并实现了一个多尺度编码器-解码器架构，以及两种特征融合模块，专门用于估算五种营养要素。这些模块通过灵活运用定制化注意力机制与融合策略，有效平衡了特征提取的效率与效能。实验研究表明，NuNet在营养估算任务上显著优于其变体及现有解决方案，其误差率低至15.65%，据我们所知此为当前最低水平，这主要得益于我们的多尺度架构与融合模块。本研究对膳食管理具有实用价值，在跨国研究与应用部署方面潜力巨大，并可为涉及多种重要性各异数据类型的其他应用提供启发。