The batch size is an essential parameter to tune during the development of new neural networks. Amongst other quality indicators, it has a large degree of influence on the model's accuracy, generalisability, training times and parallelisability. This fact is generally known and commonly studied. However, during the application phase of a deep learning model, when the model is utilised by an end-user for inference, we find that there is a disregard for the potential benefits of introducing a batch size. In this study, we examine the effect of input batching on the energy consumption and response times of five fully-trained neural networks for computer vision that were considered state-of-the-art at the time of their publication. The results suggest that batching has a significant effect on both of these metrics. Furthermore, we present a timeline of the energy efficiency and accuracy of neural networks over the past decade. We find that in general, energy consumption rises at a much steeper pace than accuracy and question the necessity of this evolution. Additionally, we highlight one particular network, ShuffleNetV2(2018), that achieved a competitive performance for its time while maintaining a much lower energy consumption. Nevertheless, we highlight that the results are model dependent.

翻译：批处理大小是开发新神经网络过程中需要调整的关键参数。在众多质量指标中，它对模型的准确性、泛化能力、训练时间和并行化程度具有显著影响。这一事实广为人知且常见于研究。然而，在深度学习模型的应用阶段（即最终用户用于推理时），我们发现引入批处理大小的潜在益处常被忽视。本研究探讨了输入批处理对五项当时被视为最先进的计算机视觉全训练神经网络在能耗和响应时间方面的影响。结果表明，批处理对这两项指标均有显著作用。此外，我们展示了过去十年神经网络能效与准确性的时间线。研究发现，能源消耗的增长速度通常远快于准确性的提升，并质疑了这一演进的必要性。同时，我们重点指出一种特定网络——ShuffleNetV2（2018），它在保持较低能耗的同时，达到了当时具有竞争力的性能。然而，我们强调结果具有模型依赖性。