Guesstimation, the task of making approximate quantity estimates, is a common real-world challenge. However, it has been largely overlooked in large language models (LLMs) and vision language models (VLMs) research. We introduce a novel guesstimation dataset, MARBLES. This dataset requires one to estimate how many items (e.g., marbles) can fit into containers (e.g., a one-cup measuring cup), both with and without accompanying images. Inspired by the social science concept of the ``Wisdom of Crowds'' (WOC) - taking the median from estimates from a crowd), which has proven effective in guesstimation, we propose ``WOC decoding'' strategy for LLM guesstimation. We show that LLMs/VLMs perform well on guesstimation, suggesting that they possess some level of a "world model" necessary for guesstimation. Moreover, similar to human performance, the WOC decoding method improves LLM/VLM guesstimation accuracy. Furthermore, the inclusion of images in the multimodal condition enhances model performance. These results highlight the value of WOC decoding strategy for LLMs/VLMs and position guesstimation as a probe for evaluating LLMs/VLMs' world model. As LLMs' world model is a fundamental prerequisite for many real-world tasks, e.g., human-AI teaming, our findings have broad implications for the AI community.

翻译：猜测估计（即进行近似数量估算的任务）是现实世界中常见的挑战。然而，在大语言模型（LLMs）和视觉语言模型（VLMs）的研究中，这一问题在很大程度上被忽视了。我们引入了一个新颖的猜测估计数据集MARBLES。该数据集要求估算有多少物品（例如弹珠）可以放入容器（例如一个一量杯的测量杯）中，包含有图像和无图像两种条件。受社会科学中“群体智慧”（WOC）概念（即取群体估计值的中位数）的启发——该概念已被证明在猜测估计中行之有效，我们提出了一种用于LLM猜测估计的“WOC解码”策略。我们证明，LLMs/VLMs在猜测估计任务上表现良好，这表明它们具备进行猜测估计所需的某种程度的“世界模型”。此外，与人类表现类似，WOC解码方法提高了LLMs/VLMs的猜测估计准确度。进一步地，在多模态条件下加入图像提升了模型性能。这些结果凸显了WOC解码策略对于LLMs/VLMs的价值，并将猜测估计定位为评估LLMs/VLMs世界模型的一种探针。由于LLMs的世界模型是许多现实世界任务（例如人机协作）的基本前提，我们的发现对人工智能领域具有广泛的启示意义。