Speech codecs serve as bridges between continuous speech signals and large language models, yet face an inherent conflict between acoustic fidelity and semantic preservation. To mitigate this conflict, prevailing methods augment acoustic codecs with complex semantic supervision. We explore the opposite direction: a semantic-first approach that starts from a semantically-capable model and adapts it for high-fidelity acoustic reconstruction. Through empirical analysis, we discover that targeted architectural simplification can unlock the acoustic modeling potential of Whisper, a text-aligned Automatic Speech Recognition (ASR) model. Based on this finding, we propose SimWhisper-Codec, a novel codec that balances the semantic and acoustic preservation by leveraging a frozen, simplified Whisper encoder without requiring external supervision. Experimental results demonstrate that SimWhisper-Codec achieves superior performance in both semantic preservation and acoustic quality compared to semantically-supervised codecs such as Mimi Codec and SpeechTokenizer at similar bitrates, validating the effectiveness of our semantic-first approach. Code is available at https://github.com/ZhangXinWhut/SimWhisper-Codec.

翻译：语音编解码器作为连续语音信号与大语言模型之间的桥梁，却面临着声学保真度与语义保持之间的固有矛盾。为缓解这一矛盾，主流方法通过复杂的语义监督来增强声学编解码器。我们探索了相反的方向：一种语义优先的方法，从具备语义能力的模型出发，并对其进行适配以实现高保真声学重建。通过实证分析，我们发现针对性的架构简化能够释放Whisper（一种文本对齐的自动语音识别模型）的声学建模潜力。基于这一发现，我们提出了SimWhisper-Codec，这是一种新颖的编解码器，它利用一个冻结的、简化后的Whisper编码器，在无需外部监督的情况下平衡了语义保持与声学保真。实验结果表明，在相似比特率下，SimWhisper-Codec在语义保持和声学质量方面均优于采用语义监督的编解码器（如Mimi Codec和SpeechTokenizer），从而验证了我们语义优先方法的有效性。代码可在https://github.com/ZhangXinWhut/SimWhisper-Codec获取。