Large Language Models (LLMs) have witnessed rapid growth in emerging challenges and capabilities of language understanding, generation, and reasoning. Despite their remarkable performance in natural language processing-based applications, LLMs are susceptible to undesirable and erratic behaviors, including hallucinations, unreliable reasoning, and the generation of harmful content. These flawed behaviors undermine trust in LLMs and pose significant hurdles to their adoption in real-world applications, such as legal assistance and medical diagnosis, where precision, reliability, and ethical considerations are paramount. These could also lead to user dissatisfaction, which is currently inadequately assessed and captured. Therefore, to effectively and transparently assess users' satisfaction and trust in their interactions with LLMs, we design and develop LLMChain, a decentralized blockchain-based reputation system that combines automatic evaluation with human feedback to assign contextual reputation scores that accurately reflect LLM's behavior. LLMChain not only helps users and entities identify the most trustworthy LLM for their specific needs, but also provides LLM developers with valuable information to refine and improve their models. To our knowledge, this is the first time that a blockchain-based distributed framework for sharing and evaluating LLMs has been introduced. Implemented using emerging tools, LLMChain is evaluated across two benchmark datasets, showcasing its effectiveness and scalability in assessing seven different LLMs.

翻译：大语言模型在语言理解、生成与推理的新兴挑战与能力方面取得了快速发展。尽管其在基于自然语言处理的应用中表现出色，但大语言模型易产生不良且不可预测的行为，包括幻觉、不可靠推理及生成有害内容。这些缺陷行为削弱了用户对大语言模型的信任，并为其在法律咨询、医疗诊断等要求精准性、可靠性与道德考量的实际应用部署设置了重大障碍。此类问题还可能导致用户不满，而当前对用户满意度的评估与捕捉尚不充分。为此，我们设计并开发了LLMChain——一种去中心化的区块链信誉系统，通过结合自动评估与人工反馈，为每个大语言模型的行为赋予上下文相关信誉评分，从而有效透明地评估用户在与大语言模型交互中的满意度与信任度。LLMChain不仅能帮助用户与实体按需识别最可信的大语言模型，还为模型开发者提供改进模型的宝贵信息。据我们所知，这是首个基于区块链的分布式大语言模型共享与评估框架。该系统采用新兴工具实现，并在两个基准数据集上对七种不同大语言模型进行了评估，验证了其有效性与可扩展性。