Virtualization-based binary obfuscation is widely adopted to protect software intellectual property, yet existing approaches leave exception-handling (EH) metadata unprotected to preserve ABI compatibility. This exposed metadata leaks rich structural information, such as stack layouts, control-flow boundaries, and object lifetimes, which can be exploited to facilitate reverse engineering. In this paper, we present XuanJia, a comprehensive VM-based binary obfuscation framework that provides end-to-end protection for both executable code and exception-handling semantics. At the core of XuanJia is ABI-Compliant EH Shadowing, a novel exception-aware protection mechanism that preserves compatibility with unmodified operating system runtimes while eliminating static EH metadata leakage. XuanJia replaces native EH metadata with ABI-compliant shadow unwind information to satisfy OS-driven unwinding, and securely redirects exception handling into a protected virtual machine where the genuine EH semantics are decrypted, reversed, and replayed using obfuscated code. We implement XuanJia from scratch, supporting 385 x86 instruction encodings and 155 VM handler templates, and design it as an extensible research testbed. We evaluate XuanJia across correctness, resilience, and performance dimensions. Our results show that XuanJia preserves semantic equivalence under extensive dynamic and symbolic testing, effectively disrupts automated reverse-engineering tools such as IDA Pro, and incurs negligible space overhead and modest runtime overhead. These results demonstrate that XuanJia achieves strong protection of exception-handling logic without sacrificing correctness or practicality.

翻译：基于虚拟化的二进制混淆技术被广泛用于保护软件知识产权，然而现有方法为保持ABI兼容性而未对异常处理元数据进行保护。这些暴露的元数据会泄露丰富的结构信息，如栈布局、控制流边界和对象生命周期，可被利用来辅助逆向工程。本文提出玄甲，一种基于虚拟机的综合性二进制混淆框架，为可执行代码和异常处理语义提供端到端保护。玄甲的核心是ABI兼容的异常处理影子化机制，这是一种新型的异常感知保护方案，在保持与未修改操作系统运行时兼容的同时，消除了静态异常处理元数据泄露。玄甲使用符合ABI规范的影子展开信息替代原生异常处理元数据，以满足操作系统驱动的栈展开需求，并将异常处理安全重定向至受保护的虚拟机中，在该虚拟机内通过混淆代码对真实的异常处理语义进行解密、逆向解析与重放。我们从头实现了玄甲系统，支持385种x86指令编码和155个虚拟机处理程序模板，并将其设计为可扩展的研究测试平台。我们从正确性、抗逆向性和性能三个维度对玄甲进行评估。实验结果表明，玄甲在广泛的动态测试与符号执行测试中保持语义等价性，能有效干扰IDA Pro等自动化逆向工程工具，且仅产生可忽略的空间开销与适度的运行时开销。这些结果证明，玄甲在不牺牲正确性或实用性的前提下，实现了对异常处理逻辑的强效保护。