Architectural Knowledge Management (AKM) involves the organized handling of information related to architectural decisions and design within a project or organization. An essential artifact of AKM is the Architecture Decision Records (ADR), which documents key design decisions. ADRs are documents that capture decision context, decision made and various aspects related to a design decision, thereby promoting transparency, collaboration, and understanding. Despite their benefits, ADR adoption in software development has been slow due to challenges like time constraints and inconsistent uptake. Recent advancements in Large Language Models (LLMs) may help bridge this adoption gap by facilitating ADR generation. However, the effectiveness of LLM for ADR generation or understanding is something that has not been explored. To this end, in this work, we perform an exploratory study that aims to investigate the feasibility of using LLM for the generation of ADRs given the decision context. In our exploratory study, we utilize GPT and T5-based models with 0-shot, few-shot, and fine-tuning approaches to generate the Decision of an ADR given its Context. Our results indicate that in a 0-shot setting, state-of-the-art models such as GPT-4 generate relevant and accurate Design Decisions, although they fall short of human-level performance. Additionally, we observe that more cost-effective models like GPT-3.5 can achieve similar outcomes in a few-shot setting, and smaller models such as Flan-T5 can yield comparable results after fine-tuning. To conclude, this exploratory study suggests that LLM can generate Design Decisions, but further research is required to attain human-level generation and establish standardized widespread adoption.

翻译：架构知识管理（AKM）涉及在项目或组织内对架构决策与设计相关信息的有组织处理。AKM的核心工件之一是架构决策记录（ADR），用于记录关键设计决策。ADR是捕捉决策背景、决策本身及与设计决策相关的多维度信息的文档，从而促进透明度、协作与理解。尽管具有诸多优势，但受限于时间约束和采用不均衡等挑战，ADR在软件开发中的普及进展缓慢。近年来大语言模型（LLM）的进展可能通过促进ADR生成来弥合这一采用差距。然而，LLM在ADR生成或理解方面的有效性尚未得到探索。为此，本研究开展了一项探索性实证研究，旨在调查在给定决策背景条件下使用LLM生成ADR的可行性。我们采用基于GPT和T5的模型，通过零样本、少样本和微调方法，根据ADR背景生成其决策内容。结果表明，在零样本设置下，GPT-4等最先进模型能生成相关且准确的设计决策，但尚未达到人类水平。此外，我们观察到更具成本效益的模型（如GPT-3.5）在少样本设置下可达到类似效果，而Flan-T5等小型模型在微调后也展现出可比性能。这项探索性研究表明LLM能够生成设计决策，但需要进一步研究才能达到人类水平的生成质量并实现标准化广泛采用。