Vision-Language-Action (VLA) models have demonstrated strong generalization in robotic manipulation, yet existing evaluations are primarily conducted in simulation or on expensive robotic platforms, leaving their robustness on affordable real-world robots largely unexplored. We present a standardized real-world benchmark for evaluating representative VLA and imitation learning policies on the low-cost SO-101 robotic platform. The benchmark comprises four representative manipulation tasks together with unified evaluation protocols, enabling systematic comparison under embodiment uncertainty. Using real-world teleoperated demonstrations, we fine-tune and evaluate $π_{0.5}$, SmolVLA, Wall-X, and ACT directly on the physical platform. Beyond conventional task success rates, the benchmark incorporates a structured failure taxonomy, semantic- and execution-level failure decomposition, and recovery-aware evaluation metrics to characterize policy robustness. Experimental results show that stronger pretrained VLA policies generally outperform the imitation learning baseline, although performance remains highly task-dependent under low-cost robotic deployment conditions. Execution instability emerges as the dominant failure source, while recovery capability varies substantially across architectures. These results highlight the importance of failure and recovery analysis beyond binary task success and establish SO-101 as a practical benchmark for evaluating embodied AI systems under realistic low-cost robotic deployment conditions.

翻译：视觉-语言-动作（VLA）模型在机器人操作中展现出强大的泛化能力，然而现有评估主要基于仿真环境或昂贵的机器人平台，其在低成本实体机器人上的鲁棒性尚待深入探究。本文提出一个标准化真实世界基准，用于在低成本SO-101机器人平台上评估代表性VLA策略与模仿学习策略。该基准包含四项典型操作任务及统一评估协议，支持在具身不确定性下的系统化比较。通过真实遥操作示范数据，我们直接在物理平台上微调并评估了$π_{0.5}$、SmolVLA、Wall-X及ACT四种模型。除传统任务成功率外，该基准引入结构化失败分类体系、语义级与执行级失败分解机制，以及基于恢复能力的评估指标，用以刻画策略鲁棒性。实验结果表明：预训练更强的VLA策略总体优于模仿学习基线，但在低成本机器人部署条件下性能呈现显著任务依赖性。执行不稳定性是主要失败来源，而恢复能力在不同架构间差异显著。这些结果凸显了超越二元任务成功率的失败与恢复分析的重要性，并将SO-101确立为在真实低成本机器人部署条件下评估具身AI系统的实用基准。