Recently, there has been growing interest within the community regarding whether large language models are capable of planning or executing plans. However, most prior studies use LLMs to generate high-level plans for simplified scenarios lacking linguistic complexity and domain diversity, limiting analysis of their planning abilities. These setups constrain evaluation methods (e.g., predefined action space), architectural choices (e.g., only generative models), and overlook the linguistic nuances essential for realistic analysis. To tackle this, we present PARADISE, an abductive reasoning task using Q\&A format on practical procedural text sourced from wikiHow. It involves warning and tip inference tasks directly associated with goals, excluding intermediary steps, with the aim of testing the ability of the models to infer implicit knowledge of the plan solely from the given goal. Our experiments, utilizing fine-tuned language models and zero-shot prompting, reveal the effectiveness of task-specific small models over large language models in most scenarios. Despite advancements, all models fall short of human performance. Notably, our analysis uncovers intriguing insights, such as variations in model behavior with dropped keywords, struggles of BERT-family and GPT-4 with physical and abstract goals, and the proposed tasks offering valuable prior knowledge for other unseen procedural tasks. The PARADISE dataset and associated resources are publicly available for further research exploration with https://github.com/GGLAB-KU/paradise.

翻译：近期，学界对大型语言模型是否具备规划或执行计划的能力日益关注。然而，先前研究多采用LLMs在简化场景中生成高层计划，缺乏语言复杂度与领域多样性，限制了对其规划能力的分析。此类设计不仅约束了评估方法（如预定义动作空间）与架构选择（如仅限生成式模型），更忽略了现实分析所需的关键语言细微差异。为解决这一问题，我们提出PARADISE——一种基于wikiHow实用程序文本的问答式溯因推理任务。该任务涉及与目标直接关联的警告与提示推断（排除中间步骤），旨在测试模型仅凭给定目标推断计划内隐知识的能力。通过微调语言模型与零样本提示的实验表明，在多数场景中任务专用小模型优于大型语言模型。尽管取得进展，所有模型仍不及人类水平。值得注意的是，我们的分析揭示了有趣现象：关键词缺失时模型行为存在差异、BERT系列与GPT-4在处理物理与抽象目标时的局限性，以及所提任务为其他未见程序任务提供的宝贵先验知识。PARADISE数据集及相关资源已开源（https://github.com/GGLAB-KU/paradise），供学界深入研究。