Ensuring high-quality data is paramount for maximizing the performance of machine learning models and business intelligence systems. However, challenges in data quality, including noise in data capture, missing records, limited data production, and confounding variables, significantly constrain the potential performance of these systems. In this study, we propose an architecture-agnostic algorithm, Gradient Guided Hypotheses (GGH), designed to address these challenges. GGH analyses gradients from hypotheses as a proxy of distinct and possibly contradictory patterns in the data. This framework entails an additional step in machine learning training, where gradients can be included or excluded from backpropagation. In this manner, missing and noisy data are addressed through a unified solution that perceives both challenges as facets of the same overarching issue: the propagation of erroneous information. Experimental validation of GGH is conducted using real-world open-source datasets, where records with missing rates of up to 98.5% are simulated. Comparative analysis with state-of-the-art imputation methods demonstrates a substantial improvement in model performance achieved by GGH. Specifically in very high scarcity regimes, GGH was found to be the only viable solution. Additionally, GGH's noise detection capabilities are showcased by introducing simulated noise into the datasets and observing enhanced model performance after filtering out the noisy data. This study presents GGH as a promising solution for improving data quality and model performance in various applications.

翻译：确保数据质量对于最大化机器学习模型与商业智能系统的性能至关重要。然而，数据质量面临的挑战——包括数据采集中的噪声、记录缺失、数据产量有限以及混杂变量——严重制约了这些系统的潜在性能。本研究提出一种与架构无关的算法，即梯度引导假设（GGH），旨在应对这些挑战。GGH通过分析假设梯度作为数据中不同且可能矛盾模式的代理指标。该框架在机器学习训练中引入了一个额外步骤，使得梯度可被选择性地纳入或排除于反向传播过程。通过这种方式，缺失数据与噪声数据问题通过统一方案得以处理，该方案将二者视为同一核心问题的不同表现：错误信息的传播。GGH的实验验证采用真实世界开源数据集进行，其中模拟了缺失率高达98.5%的记录。与前沿插补方法的对比分析表明，GGH实现了模型性能的显著提升。特别是在极高数据稀缺场景下，GGH被证明是唯一可行的解决方案。此外，通过在数据集中引入模拟噪声并观察滤除噪声数据后模型性能的提升，展示了GGH的噪声检测能力。本研究证明GGH是一种能够提升多领域应用数据质量与模型性能的潜力解决方案。