The CPU-side large language model (LLM) tokenizer is a critical security gap in LLM serving through a confidential computing stack with CPU and GPU trusted execution environments (TEEs). Tokenizers converts the prompts through table-driven lookups, and the resulting memory access patterns are a powerful source of side-channel leakage. Recent work demonstrates end-to-end recovery of user prompts from tokenizer access pattern on production Intel TDX. However, a drop-in use of the popular tree-based Oblivious RAMs (e.g., PathORAM) to prevent access-pattern leakage introduces $\sim$13$\times$ tokenizer slowdown, resulting in 10-58% higher time-to-first-token (TTFT). In this paper, we present OTRO, an efficient, oblivious tokenization path tailored to latency-critical LLM serving. OTRO relies on square-root ORAM for fast single-access lookups, but avoids its prohibitive $O(N\log^2N$) rebuild cost every $\sqrt{N}$ accesses through three key innovations. First, OTRO provides a pool of replicated square-root ORAM instances that utilize the read-only nature of tokenizer table. Second, an epoch-based rotation policy decouples accesses from rebuilds and pads each epoch with dummy accesses to its boundaries, minimizing observable information. Lastly, chunked KV-cache-aware tokenization further overlaps rebuilds with GPU prefill and minimizes the instance count. Implemented as modules in HuggingFace Tokenizers and nano-vLLM, running within a TDX-enabled CVM with an NVIDIA H100 GPU, OTRO limits TTFT overhead to at most 4.5%, keeps tokenizer-induced latency under 10\% of total TTFT, and adds less than 0.5 GB of memory overhead while reducing the tokenizer's observable leakage across various model families and sizes.

翻译：CPU端大型语言模型（LLM）分词器是LLM通过CPU与GPU可信执行环境（TEE）机密计算栈提供服务时的关键安全缺口。分词器通过查表方式转换提示词，其产生的内存访问模式构成强大的侧信道泄露源。近期研究已展示在Intel TDX生产环境中，可通过分词器访问模式端到端恢复用户提示词。然而，直接采用基于树的流行不经意随机访问机（如PathORAM）来防止访问模式泄露会引入约13倍的分词器性能下降，导致首令牌生成时间（TTFT）增加10-58%。本文提出OTRO，一种面向延迟敏感的LLM服务的高效不经意的令牌化路径。OTRO采用平方根ORAM实现快速单次访问查找，但通过三项关键创新避免了每次√N次访问时高达O(N log²N)的重建代价。首先，OTRO提供复制的平方根ORAM实例池，利用分词器表的只读特性。其次，基于时段的旋转策略将访问与重建解耦，并通过假访问填充各时段的边界，最小化可观测信息。最后，分块KV缓存感知的令牌化进一步将重建与GPU预填充重叠，并最小化实例数量。作为HuggingFace Tokenizers和nano-vLLM的模块实现，在搭载NVIDIA H100 GPU的TDX虚拟机中运行时，OTRO将TTFT开销限制在4.5%以内，分词器引发的延迟不超过总TTFT的10%，内存开销低于0.5GB，同时在不同模型族和规模下有效降低了分词器的可观测泄露。