Large Language Models (LLMs) have been applied to build several automation and personalized question-answering prototypes so far. However, scaling such prototypes to robust products with minimized hallucinations or fake responses still remains an open challenge, especially in niche data-table heavy domains such as financial decision making. In this work, we present a novel Langchain-based framework that transforms data tables into hierarchical textual data chunks to enable a wide variety of actionable question answering. First, the user-queries are classified by intention followed by automated retrieval of the most relevant data chunks to generate customized LLM prompts per query. Next, the custom prompts and their responses undergo multi-metric scoring to assess for hallucinations and response confidence. The proposed system is optimized with user-query intention classification, advanced prompting, data scaling capabilities and it achieves over 90% confidence scores for a variety of user-queries responses ranging from {What, Where, Why, How, predict, trend, anomalies, exceptions} that are crucial for financial decision making applications. The proposed data to answers framework can be extended to other analytical domains such as sales and payroll to ensure optimal hallucination control guardrails.

翻译：大型语言模型（LLMs）已被应用于构建多种自动化和个性化问答原型。然而，将这些原型扩展为具有最小化幻觉或虚假响应的稳健产品，仍是一个公开挑战，尤其是在金融决策等数据表格密集的利基领域。本文提出了一种基于Langchain的新型框架，该框架将数据表格转化为分层文本数据块，以支持多种可行的问答任务。首先，用户查询按意图进行分类，随后自动检索最相关的数据块，为每个查询生成定制的LLM提示。接着，定制提示及其响应经过多指标评分，以评估幻觉和响应置信度。该框架通过用户查询意图分类、高级提示工程以及数据扩展能力进行优化，在为金融决策应用至关重要的{什么、哪里、为什么、如何、预测、趋势、异常、例外}等各类用户查询响应中，实现了超过90%的置信度分数。所提出的数据到答案框架可扩展至销售和薪资等其他分析领域，以确保最佳的幻觉控制护栏。