In process tracing, researchers ask how strongly their evidence favors their explanation, the working theory, over a rival. Fairfield and Charman (2022) compare the two theories with a Bayes factor whose probabilities are set by hand, which Zaks (2021) argues lets researchers overstate findings. We derive those probabilities from a fully specified model of the evidence a researcher could have examined, given which observations support each theory. The working theory says most of the evidence supports it. The rival says no more than half does. The model gives two Bayes factors, neither designed to favor the working theory. The second grants the rival more benefit of the doubt and is provably the smallest value the rival's claim allows. Researchers can report how much observation bias, re-coding, or a smoking-gun weight would change a conclusion. In six published studies we reanalyze, both Bayes factors exceed 20 for two studies and not for the other four.
翻译:暂无翻译