Dynamic adversarial question generation, where humans write examples to stump a model, aims to create examples that are realistic and informative. However, the advent of large language models (LLMs) has been a double-edged sword for human authors: more people are interested in seeing and pushing the limits of these models, but because the models are so much stronger an opponent, they are harder to defeat. To understand how these models impact adversarial question writing process, we enrich the writing guidance with LLMs and retrieval models for the authors to reason why their questions are not adversarial. While authors could create interesting, challenging adversarial questions, they sometimes resort to tricks that result in poor questions that are ambiguous, subjective, or confusing not just to a computer but also to humans. To address these issues, we propose new metrics and incentives for eliciting good, challenging questions and present a new dataset of adversarially authored questions.

翻译：动态对抗问题生成旨在通过人类撰写示例以难倒模型，从而创建既真实又富含信息的问题。然而，大语言模型（LLMs）的出现对人类作者而言是一把双刃剑：更多人热衷于观察并挑战这些模型的极限，但由于模型作为对手变得异常强大，人类更难将其击败。为理解这些模型如何影响对抗问题的撰写过程，我们为作者补充了基于LLMs和检索模型的写作指导，以帮助其分析为何他们提出的问题不具备对抗性。尽管作者能创造出有趣且具挑战性的对抗问题，但有时会采用技巧性手段，导致生成的问题存在歧义、主观性强或混淆性——不仅对计算机如此，对人类也同样如此。为解决这些问题，我们提出了新的评估指标与激励机制，以引导生成高质量、高难度的问题，并推出一个全新的对抗性作者问题数据集。