LLMs are powerful generators of synthetic data, which are used for training smaller, specific models. This is especially valuable for low-resource languages, where human-labelled data is scarce but LLMs can still produce high-quality text. However, LLMs differ in how useful their outputs are for training. Selecting the best LLM as a generator is challenging because extrinsic evaluation requires costly human annotations (which are often unavailable for low-resource languages), while intrinsic metrics correlate poorly with downstream performance. We introduce Round robin Synthetic data Evaluation (RoSE), a proxy metric for selecting the best LLM generator without human test sets. RoSE trains a small model on the outputs of a candidate generator (LLM) and then evaluates it on generated synthetic examples from all other candidate LLMs. The final RoSE score is the mean performance of this small model. Across six LLMs, eleven languages, and three tasks (sentiment, topic, intent), RoSE identifies the optimal generator more often than any other intrinsic heuristics. RoSE outperforms intrinsic heuristics and comes within 0.76 percentage points of the optimal generator baseline. This result is measured in terms of downstream performance, obtained by training a small model on the chosen generator's outputs (optimal vs. proxy metric selected) and evaluating it on human-labelled test data. Additionally, RoSE is the only metric to achieve a positive correlation with performance on human test data.

翻译：大语言模型（LLM）是合成数据的强大生成器，这些数据可用于训练更小型的专用模型。这对于低资源语言尤为宝贵，因为此类语言中人工标注数据稀缺，但LLM仍能生成高质量文本。然而，不同LLM生成的输出在训练中具有不同的实用价值。由于外部评估需要昂贵的人工标注（这在低资源语言场景中通常难以获取），而内部指标与下游性能的相关性较弱，因此选择最佳LLM作为生成器颇具挑战。我们提出轮询合成数据评估方法（RoSE），这是一种无需人工测试集即可选择最佳LLM生成器的代理指标。RoSE将候选生成器（LLM）的输出用于训练小型模型，并基于其他所有候选LLM生成的合成样本对该模型进行评估，最终RoSE分数即为该小模型的平均性能。在六个LLM、十一种语言及三类任务（情感分析、主题分类、意图识别）中，RoSE比任何其他内部启发式方法更频繁地识别出最优生成器。RoSE的性能优于内部启发式方法，与最优生成器基线的差距仅为0.76个百分点。该结果通过下游性能指标度量：即基于所选生成器的输出（最优生成器vs代理指标选定生成器）训练小型模型，并在人工标注测试数据上评估。此外，RoSE是唯一与人工测试数据性能呈正相关的评估指标。