面向物理 GUI 交互测试的多模态机器人框架
原标题：A Multi-Modal Robotic Framework for Physical GUI Interaction Testing
AI 摘要
该博士研究提出 RFNIT，一个多模态、完全非侵入式的机器人框架，用于在智能手机上进行物理 GUI 交互测试。框架结合基于视觉的感知模块、LLM 驱动的推理智能体和可执行触摸、滑动、打字手势的机器人执行器，通过外部摄像头图像与 OCR 文本识别 GUI 元素。初步结果显示其对外部摄像头输入中的按钮、文本框和虚拟键盘识别准确率超过 85%，未来将提升鲁棒性、可重复性和长时间测试的 GUI 状态覆盖率。
正文节选
A Multi-Modal Robotic Framework for Physical GUI Interaction Testing Abstract. Automated testing of mobile applications remains challenging when realistic user interactions must be reproduced and GUI behavior verified without relying on invasive mechanisms such as instrumentation or ADB-level control. Existing approaches, including DLD and R-DLD, detect data loss triggered by activity restarts but still depend on system-level access, which limits their applicability on real devices. This doctora