Editing pretrained neural networks requires specialized algorithms tailored to specific objectives. Designing such algorithms is often time-consuming and demands significant effort. We present an exploratory framework that formulates neural model editing as a reinforcement learning problem, where agents modify models using reward feedback. We introduce two environments: MaskWorld, where agents scale weights multiplicatively, and ShiftWorld, where agents apply additive weight updates. The reward function combines a utility-preservation objective with a task-specific editing objective, enabling agents to learn targeted modifications while maintaining overall model performance. We evaluate the framework on bias mitigation in text classification and machine unlearning in image classification, both of which traditionally rely on specialized algorithms. Our results show that the learned policies reduce forget set accuracy to nearly 0% while preserving over 90% retain set accuracy on the unlearning task. In the bias mitigation setting, the learned policies improve bias-related performance by more than 5% while maintaining general classification utility. Our findings show that neural model editing can be cast as a reinforcement learning problem, allowing editing policies to be learned from reward feedback rather than manually engineered for each task.

翻译：编辑预训练神经网络需要针对特定目标定制的专用算法，这类算法的设计过程通常耗时且工作量巨大。我们提出一个探索性框架，将神经模型编辑形式化为强化学习问题，智能体通过奖励反馈修改模型。我们引入两个环境：MaskWorld（智能体以乘法方式缩放权重）与ShiftWorld（智能体进行加法权重更新）。奖励函数结合了效用保持目标与任务特定编辑目标，使智能体在学习针对性修改的同时维持模型整体性能。我们在文本分类的偏差缓解与图像分类的机器遗忘任务上评估该框架，这两类任务传统上均依赖专用算法。结果表明，在遗忘任务中，学习到的策略可将遗忘集准确率降至接近0%，同时保留集准确率保持在90%以上。在偏差缓解场景中，学习到的策略将偏差相关性能提升超过5%，同时维持通用分类效用。研究证实神经模型编辑可构建为强化学习问题，使编辑策略能通过奖励反馈习得，无需为每个任务手动设计算法。