This paper proposes a method for automatic GUI component detection for the IBM i system (formerly and still more commonly known as AS/400). We introduce a human-annotated dataset consisting of 1,050 system screen images, in which 381 images are screenshots of IBM i system screens in Japanese. Each image contains multiple components, including text labels, text boxes, options, tables, instructions, keyboards, and command lines. We then develop a detection system based on state-of-the-art deep learning models and evaluate different approaches using our dataset. The experimental results demonstrate the effectiveness of our dataset in constructing a system for component detection from GUI screens. By automatically detecting GUI components from the screen, AS400-DET has the potential to perform automated testing on systems that operate via GUI screens.

翻译：本文提出了一种针对IBM i系统（原名且更常被称为AS/400）的自动图形用户界面组件检测方法。我们引入了一个包含1,050张系统屏幕图像的人工标注数据集，其中381张为日文版IBM i系统屏幕截图。每张图像包含多个组件，如文本标签、文本框、选项、表格、说明、键盘及命令行。随后，我们基于前沿的深度学习模型开发了一个检测系统，并利用我们的数据集评估了不同方法。实验结果表明，我们的数据集在构建面向GUI屏幕的组件检测系统方面具有有效性。通过自动从屏幕中检测GUI组件，AS400-DET有望对通过GUI屏幕操作的系统执行自动化测试。