The integration of large-scale chemical databases represents a critical bottleneck in modern cheminformatics research, particularly for machine learning applications requiring high-quality, multi-source validated datasets. This paper presents a case study of integrating three major public chemical repositories: PubChem (176 million compounds), ChEMBL, and eMolecules, to construct a curated dataset for molecular property prediction. We investigate whether byte-offset indexing can practically overcome brute-force scalability limits while preserving data integrity at hundred-million scale. Our results document the progression from an intractable brute-force search algorithm with projected 100-day runtime to a byte-offset indexing architecture achieving 3.2-hour completion - a 740-fold performance improvement through algorithmic complexity reduction from $O(N \times M)$ to $O(N + M)$. Systematic validation of 176 million database entries revealed hash collisions in InChIKey molecular identifiers, necessitating pipeline reconstruction using collision-free full InChI strings. We present performance benchmarks, quantify trade-offs between storage overhead and scientific rigor, and compare our approach with alternative large-scale integration strategies. The resulting system successfully extracted 435,413 validated compounds and demonstrates generalizable principles for large-scale scientific data integration where uniqueness constraints exceed hash-based identifier capabilities.

翻译：大规模化学数据库的整合是当代化学信息学研究中的关键瓶颈，尤其对于需要高质量、多源验证数据集的机器学习应用而言。本文以三大公共化学资源库——PubChem（1.76亿化合物）、ChEMBL和eMolecules的整合为案例，构建了一个用于分子性质预测的精选数据集。我们探究了字节偏移索引能否在十亿级数据规模下实际突破暴力扩展限制，同时保持数据完整性。研究结果记录了从预计运行100天的不可行暴力搜索算法，到仅需3.2小时完成的字节偏移索引架构的演进过程——通过将算法复杂度从$O(N \times M)$降至$O(N + M)$，实现了740倍的性能提升。对1.76亿数据库条目的系统性验证揭示了InChIKey分子标识符存在哈希碰撞现象，因此需要采用无碰撞的完整InChI字符串重建数据流水线。我们提供了性能基准测试结果，量化了存储开销与科学严谨性之间的权衡，并将本方案与其他大规模整合策略进行了对比。最终系统成功提取了435,413个已验证化合物，并展示了在唯一性约束超出哈希标识符能力时，适用于大规模科学数据整合的可推广原则。