When a large language model (LLM) writes Harry Potter fanfiction, it can produce fundamental elements of the Hogwarts universe, such as recognizable places and characters. Human-written Harry Potter fanfiction, however, typically includes these fundamentals and much more, incorporating stylistically irregular content and relationship-diverse plotlines. This gap between LLM and human writing has been noted across a variety of domains. LLMs tend to produce "average" writing, while human writing contains more diverse content that covers a broader distribution. Existing work has shown the existence of this distributional "gap". We take a further step by providing a systematic framework to measure coverage of the human response distribution and identify which regions model samples reach or miss. Our paper proposes a human-grounded framework that uses the empirical distribution of human writing on a topic to measure the distributional breadth of LLM-generated content on that same topic. We propose two metrics, LLM Coverage (LLM-Cov) and In-Boundary Rate (IBR), which measure both distributional breadth and membership in the human reference region, respectively. Across ideation and narrative tasks, we find that the tested LLMs have lower mean coverage than equally sized held-out human samples. We also find that model reach is disproportionately concentrated near the center of the human response space relative to this human benchmark. In Harry Potter fanfiction, covered and uncovered passages also show systematic differences in canon anchoring and narrative form. Our framework can enable researchers to better assess the distributional breadth--what we term the "cultural reach"--of LLM-authored content.
翻译:暂无翻译