The integration of large-scale chemical databases represents a critical bottleneck in modern cheminformatics research, particularly for machine learning applications requiring high-quality, multi-source validated datasets. This paper presents a case study of integrating three major public chemical repositories: PubChem (176 million compounds), ChEMBL, and eMolecules, to construct a curated dataset for molecular property prediction. We investigate whether byte-offset indexing can practically overcome brute-force scalability limits while preserving data integrity at hundred-million scale. Our results document the progression from an intractable brute-force search algorithm with projected 100-day runtime to a byte-offset indexing architecture achieving 3.2-hour completion-a 740-fold performance improvement through algorithmic complexity reduction from O(NxM) to O(N+M). Systematic validation of 176 million database entries revealed hash collisions in InChIKey molecular identifiers, necessitating pipeline reconstruction using collision-free full InChI strings. We present performance benchmarks, quantify trade-offs between storage overhead and scientific rigor, and compare our approach with alternative large-scale integration strategies. The resulting system successfully extracted 435,413 validated compounds and demonstrates generalizable principles for large-scale scientific data integration where uniqueness constraints exceed hash-based identifier capabilities.

翻译：大规模化学数据库的整合是现代化学信息学研究的关键瓶颈，尤其在机器学习应用需要高质量、多源验证数据集的背景下。本文以整合三大公共化学资源库——PubChem（1.76亿化合物）、ChEMBL和eMolecules——构建用于分子性质预测的精选数据集为案例进行研究。我们探讨了字节偏移索引是否能在保持亿级数据完整性的同时，实际克服暴力搜索的可扩展性限制。研究结果记录了从预计需100天运行时间的不可行暴力搜索算法，到采用字节偏移索引架构实现3.2小时完成的演进过程——通过算法复杂度从O(NxM)降至O(N+M)，性能提升达740倍。对1.76亿条数据库记录的系统验证揭示了InChIKey分子标识符中的哈希冲突，这要求我们使用无冲突的完整InChI字符串重建处理流程。我们提供了性能基准测试，量化了存储开销与科学严谨性之间的权衡，并将本方法与替代性大规模整合策略进行比较。最终构建的系统成功提取了435,413个已验证化合物，并为在唯一性约束超出基于哈希的标识符能力时的大规模科学数据整合提供了可推广的原则。