Large Language Models (LLMs) have garnered significant attention for their ability to understand text and images, generate human-like text, and perform complex reasoning tasks. However, their ability to generalize this advanced reasoning with a combination of natural language text for decision-making in dynamic situations requires further exploration. In this study, we investigate how well LLMs can adapt and apply a combination of arithmetic and common-sense reasoning, particularly in autonomous driving scenarios. We hypothesize that LLMs hybrid reasoning abilities can improve autonomous driving by enabling them to analyze detected object and sensor data, understand driving regulations and physical laws, and offer additional context. This addresses complex scenarios, like decisions in low visibility (due to weather conditions), where traditional methods might fall short. We evaluated Large Language Models (LLMs) based on accuracy by comparing their answers with human-generated ground truth inside CARLA. The results showed that when a combination of images (detected objects) and sensor data is fed into the LLM, it can offer precise information for brake and throttle control in autonomous vehicles across various weather conditions. This formulation and answers can assist in decision-making for auto-pilot systems.

翻译：大型语言模型（LLMs）因其理解文本与图像、生成类人文本以及执行复杂推理任务的能力而备受关注。然而，它们能否将这种高级推理能力与自然语言文本结合，在动态场景中进行决策，仍需进一步探索。本研究考察了LLMs如何适应并综合运用算术推理与常识推理，特别是在自动驾驶场景中的应用。我们假设，LLMs的混合推理能力能够通过分析检测到的物体与传感器数据、理解交通法规与物理定律、并提供额外上下文信息，从而提升自动驾驶性能。这可以解决传统方法可能失效的复杂场景（如因天气原因导致的低能见度决策）。我们基于准确率评估了LLMs的性能，将其回答与CARLA仿真器中人工生成的基准答案进行对比。结果表明，当将图像（检测到的物体）与传感器数据共同输入LLM时，该模型能精确提供自动驾驶车辆在各种天气条件下制动与油门控制所需的决策信息。这种推理框架与输出结果可为自动驾驶系统的决策提供辅助。