A small transformer encoder is trained to map Terminal-Wrench trajectories onto a unit sphere where embedding distance approximates the $L_1$ distance between reward and metadata signals. A linear probe on top of that embedding detects reward hacking on the cleaned test split with AUC $0.9467$ and TPR@5%FPR $0.8296$, matching the TW sanitized LLM-as-judge AUC ($0.9510$ on the cleaned split) and exceeding its TPR@5%FPR ($0.7130$ vs $0.8296$) on the same information condition, at roughly four orders of magnitude lower per-trajectory cost. The encoder is not a pure behavior reader: stripping natural-language reasoning from its input at probe time drops AUC to $0.6213$.
翻译:一个小型Transformer编码器被训练用于将Terminal-Wrench轨迹映射到单位球面上,使得嵌入距离近似于奖励和元数据信号之间的$L_1$距离。在该嵌入之上添加的线性探测头能在清洗后的测试集上以AUC $0.9467$和TPR@5%FPR $0.8296$检测到奖励篡改,在相同信息条件下匹配TW净化的LLM-as-judge的AUC(清洗后集上为$0.9510$)并超越其TPR@5%FPR($0.7130$对比$0.8296$),而每条轨迹的计算成本降低了约四个数量级。该编码器并非纯粹的行为解读器:在探测时从其输入中剥离自然语言推理会使AUC降至$0.6213$。