Query formulation from internal information needs remains fundamentally challenging across all Information Retrieval paradigms due to cognitive complexity and physical impairments. Brain Passage Retrieval (BPR) addresses this by directly mapping EEG signals to passage representations without intermediate text translation. However, existing BPR research exclusively uses visual stimuli, leaving critical questions unanswered: Can auditory EEG enable effective retrieval for voice-based interfaces and visually impaired users? Can training on combined EEG datasets from different sensory modalities improve performance despite severe data scarcity? We present the first systematic investigation of auditory EEG for BPR and evaluate cross-sensory training benefits. Using dual encoder architectures with four pooling strategies (CLS, mean, max, multi-vector), we conduct controlled experiments comparing auditory-only, visual-only, and combined training on the Alice (auditory) and Nieuwland (visual) datasets. Results demonstrate that auditory EEG consistently outperforms visual EEG, and cross-sensory training with CLS pooling achieves substantial improvements over individual training: 31% in MRR (0.474), 43% in Hit@1 (0.314), and 28% in Hit@10 (0.858). Critically, combined auditory EEG models surpass BM25 text baselines (MRR: 0.474 vs 0.428), establishing neural queries as competitive with traditional retrieval whilst enabling accessible interfaces. These findings validate auditory neural interfaces for IR tasks and demonstrate that cross-sensory training addresses data scarcity whilst outperforming single-modality approaches Code: https://github.com/NiallMcguire/Audio_BPR

翻译：基于内在信息需求构建查询始终是信息检索范式中面临的根本性挑战，其根源在于认知复杂性和生理机能障碍。脑部段落检索通过直接建立脑电图信号与段落表示之间的映射关系，绕过了中间文本翻译环节。然而现有脑部段落检索研究仅采用视觉刺激，留下若干关键问题亟待解答：听觉脑电图能否为语音交互界面及视障用户提供有效检索？尽管数据极度匮乏，组合不同感觉通道的脑电图数据集进行训练能否提升性能？本文首次系统性地研究了听觉脑电图用于脑部段落检索的可行性，并评估了跨感官联合训练的优势。采用双编码器架构及四种池化策略（CLS、均值池化、最大池化、多向量池化），我们在Alice（听觉）和Nieuwland（视觉）数据集上开展了听觉单模态、视觉单模态及跨模态联合训练的对比实验。结果表明：听觉脑电图检索性能持续优于视觉脑电图；采用CLS池化的跨感官训练相较于单模态训练实现显著提升——MRR提升31%（0.474）、Hit@1提升43%（0.314）、Hit@10提升28%（0.858）。尤为关键的是，听觉脑电图联合模型超越了BM25文本基线（MRR：0.474 vs 0.428），在保持与传统检索竞争力度的同时拓展了无障碍交互界面的可能性。这些发现验证了听觉神经接口在信息检索任务中的有效性，并证明跨感官训练既能解决数据稀缺问题，其性能又优于单模态方法。代码地址：https://github.com/NiallMcguire/Audio_BPR