Qualitative research faces a critical reliability challenge: traditional inter-rater agreement methods require multiple human coders, are time-intensive, and often yield moderate consistency. We present a multi-perspective validation framework for LLM-based thematic analysis that combines ensemble validation with dual reliability metrics: Cohen's Kappa ($κ$) for inter-rater agreement and cosine similarity for semantic consistency. Our framework enables configurable analysis parameters (1-6 seeds, temperature 0.0-2.0), supports custom prompt structures with variable substitution, and provides consensus theme extraction across any JSON format. As proof-of-concept, we evaluate three leading LLMs (Gemini 2.5 Pro, GPT-4o, Claude 3.5 Sonnet) on a psychedelic art therapy interview transcript, conducting six independent runs per model. Results demonstrate Gemini achieves highest reliability ($κ= 0.907$, cosine=95.3%), followed by GPT-4o ($κ= 0.853$, cosine=92.6%) and Claude ($κ= 0.842$, cosine=92.1%). All three models achieve a high agreement ($κ> 0.80$), validating the multi-run ensemble approach. The framework successfully extracts consensus themes across runs, with Gemini identifying 6 consensus themes (50-83% consistency), GPT-4o identifying 5 themes, and Claude 4 themes. Our open-source implementation provides researchers with transparent reliability metrics, flexible configuration, and structure-agnostic consensus extraction, establishing methodological foundations for reliable AI-assisted qualitative research.

翻译：定性研究面临一个关键的可靠性挑战：传统评分者间一致性方法需要多位人工编码员，耗时费力，且通常只能产生中等程度的一致性。我们提出一个用于基于LLM的主题分析的多视角验证框架，该框架将集成验证与双可靠性度量相结合：用于评分者间一致性的Cohen's Kappa ($κ$) 和用于语义一致性的余弦相似度。我们的框架支持可配置的分析参数（1-6个种子，温度0.0-2.0），支持带有变量替换的自定义提示结构，并提供跨任何JSON格式的共识主题提取。作为概念验证，我们在一个迷幻艺术疗法访谈记录上评估了三种领先的LLM（Gemini 2.5 Pro, GPT-4o, Claude 3.5 Sonnet），每个模型进行六次独立运行。结果表明，Gemini实现了最高的可靠性（$κ= 0.907$，余弦相似度=95.3%），其次是GPT-4o（$κ= 0.853$，余弦相似度=92.6%）和Claude（$κ= 0.842$，余弦相似度=92.1%）。所有三种模型均达到了高度一致性（$κ> 0.80$），验证了多轮集成方法的有效性。该框架成功提取了跨运行轮次的共识主题，其中Gemini识别出6个共识主题（一致性50-83%），GPT-4o识别出5个主题，Claude识别出4个主题。我们的开源实现为研究人员提供了透明的可靠性度量、灵活的配置以及与结构无关的共识提取，为可靠的AI辅助定性研究奠定了方法论基础。