Large Language Models (LLMs) have garnered significant attention for their ability to understand text and images, generate human-like text, and perform complex reasoning tasks. However, their ability to generalize this advanced reasoning with a combination of natural language text for decision-making in dynamic situations requires further exploration. In this study, we investigate how well LLMs can adapt and apply a combination of arithmetic and common-sense reasoning, particularly in autonomous driving scenarios. We hypothesize that LLMs hybrid reasoning abilities can improve autonomous driving by enabling them to analyze detected object and sensor data, understand driving regulations and physical laws, and offer additional context. This addresses complex scenarios, like decisions in low visibility (due to weather conditions), where traditional methods might fall short. We evaluated Large Language Models (LLMs) based on accuracy by comparing their answers with human-generated ground truth inside CARLA. The results showed that when a combination of images (detected objects) and sensor data is fed into the LLM, it can offer precise information for brake and throttle control in autonomous vehicles across various weather conditions. This formulation and answers can assist in decision-making for auto-pilot systems.

翻译：大语言模型（LLMs）因其理解文本与图像、生成类人文本以及执行复杂推理任务的能力而备受关注。然而，其在动态场景中结合自然语言文本进行决策时泛化高级推理的能力仍需深入探究。本研究考察了大语言模型在自主驾驶场景中适配并运用算术与常识混合推理的能力。我们假设大语言模型的混合推理能力能够通过分析检测目标与传感器数据、理解驾驶法规与物理定律、并提供额外上下文来提升自主驾驶性能。这有助于应对传统方法可能失效的复杂场景（如恶劣天气导致的低能见度决策）。我们基于CARLA模拟器中人类生成的基准真值，通过精确度指标评估了大语言模型的表现。结果表明，当将图像（检测目标）与传感器数据联合输入大语言模型时，该模型能在不同天气条件下为自主车辆的制动与油门控制提供精确信息。上述建模方法与输出结果可为自动驾驶系统的决策提供支撑。