Large Language Models (LLMs) are increasingly deployed in high-stakes contexts where their outputs influence real-world decisions. However, evaluating bias in LLM outputs remains methodologically challenging due to sensitivity to prompt wording, limited multilingual coverage, and the lack of standardized metrics that enable reliable comparison across models. This paper introduces BiasLab, an open-source, model-agnostic evaluation framework for quantifying output-level (extrinsic) bias through a multilingual, robustness-oriented experimental design. BiasLab constructs mirrored probe pairs under a strict dual-framing scheme: an affirmative assertion favoring Target A and a reverse assertion obtained by deterministic target substitution favoring Target B, while preserving identical linguistic structure. To reduce dependence on prompt templates, BiasLab performs repeated evaluation under randomized instructional wrappers and enforces a fixed-choice Likert response format to maximize comparability across models and languages. Responses are normalized into agreement labels using an LLM-based judge, aligned for polarity consistency across framings, and aggregated into quantitative bias indicators with descriptive statistics including effect sizes and neutrality rates. The framework supports evaluation across diverse bias axes, including demographic, cultural, political, and geopolitical topics, and produces reproducible artifacts such as structured reports and comparative visualizations. BiasLab contributes a standardized methodology for cross-lingual and framing-sensitive bias measurement that complements intrinsic and dataset-based audits, enabling researchers and institutions to benchmark robustness and make better-informed deployment decisions.

翻译：大语言模型（LLMs）正日益部署于高风险场景中，其输出会影响现实世界的决策。然而，由于对提示措辞的敏感性、多语言覆盖范围有限以及缺乏能够实现模型间可靠比较的标准化指标，评估LLM输出中的偏见在方法论上仍具挑战性。本文介绍BiasLab，这是一个开源的、模型无关的评估框架，旨在通过一个多语言的、以稳健性为导向的实验设计来量化输出级（外在）偏见。BiasLab在严格的双框架方案下构建镜像探测对：一个支持目标A的肯定性断言，以及一个通过确定性目标替换获得的、支持目标B的反向断言，同时保持完全相同的语言结构。为了减少对提示模板的依赖，BiasLab在随机化的指令包装下进行重复评估，并强制执行固定选择的Likert响应格式，以最大化模型和语言间的可比性。响应通过基于LLM的评判器归一化为一致性标签，并根据框架进行极性一致性对齐，然后聚合成带有描述性统计（包括效应量和中立率）的定量偏见指标。该框架支持跨多种偏见维度的评估，包括人口统计学、文化、政治和地缘政治主题，并生成可复现的成果，如结构化报告和比较可视化图表。BiasLab提供了一种标准化的方法，用于跨语言和对框架敏感的偏见测量，补充了内在和基于数据集的审计，使研究人员和机构能够对模型的稳健性进行基准测试，并做出更明智的部署决策。