增强现实辅助的互认知人机安全交互系统

doi:10.3901/JME.2023.06.173

机械工程学报 ›› 2023, Vol. 59 ›› Issue (6): 173-184.doi: 10.3901/JME.2023.06.173

扫码分享

增强现实辅助的互认知人机安全交互系统

郑湃^1,2, 李成熙¹, 殷悦¹, 张荣³, 鲍劲松³, 王柏村⁴, 谢海波⁴, 王力翚⁵

1. 香港理工大学工业及系统工程系香港 999077;
2. 香港理工大学深圳研究院深圳 518057;
3. 东华大学机械工程学院上海 201620;
4. 浙江大学机械工程学院杭州 310058;
5. 瑞典皇家理工学院斯德哥尔摩 SE-11428 瑞典

收稿日期:2022-03-29 修回日期:2022-12-21 出版日期:2023-03-20 发布日期:2023-06-03
通讯作者: 郑湃(通信作者),男,1988年出生,博士,助理教授,博士研究生导师。主要研究方向为人机协作制造,智能产品服务系统和工业数字化服务。E-mail:pai.zheng@polyu.edu.hk
作者简介:李成熙,男,1994年出生,博士研究生。主要研究方向为强化学习,机器人学习,增强/虚拟现实,多智能体制造系统。E-mail:chengxi.li@connect.polyu.hk
基金资助:
国家自然科学基金(52005424, 52205542)、香港研究资助局General Research Fund (15210222)、香港创新科技署AIR@InnoHK创新平台 (AiDLab-RP2-1)和江苏省政策引导类计划(港澳台科技合作)(BZ2020049)资助项目。

Augmented Reality-assisted Mutual Cognitive System for Human-Robot Interaction Safety Concerns

ZHENG Pai^1,2, LI Chengxi¹, YIN Yue¹, ZHANG Rong³, BAO Jinsong³, WANG Baicun⁴, XIE Haibo⁴, WANG Lihui⁵

1. Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University, Hong Kong 999077;
2. The Hong Kong Polytechnic University Shenzhen Research Institute, Shenzhen 518057;
3. College of Mechanical Engineering, Donghua University, Shanghai 201620;
4. School of Mechanical Engineering, Zhejiang University, Hangzhou 310058;
5. KTH Royal Institute of Technology, Stockholm SE-11428, Sweden

Received:2022-03-29 Revised:2022-12-21 Online:2023-03-20 Published:2023-06-03

摘要/Abstract

摘要： 在现代制造业中，人和机器人的交互共融是制造智能化的焦点问题之一。在人机交互中，机器人对工人的伤害风险是影响安全生产的关键因素。然而，在动态、不确定性的人机交互环境中，目前的人机安全交互系统仍然是基于机器人对环境的感知实现被动避让，缺乏基于互认知的自适应决策。为提高人对环境的认知能力，并增强机器人的自主避让和自适应能力，设计并实现了一种基于可穿戴增强现实技术的互认知人机安全交互系统。其中，可穿戴增强现实设备作为人机交互的媒介，实现机器人的虚实注册和工作环境的虚拟映射，收集人、机器人和作业空间的信息，实时为操作人员和机器人提供互认知辅助。互认知人机安全交互系统从三个方面确保人机安全：根据人机距离的机器人速度控制和安全区域可视化；虚实映射的运动预览和碰撞检测；基于深度强化学习的机器人避障策略。最后，通过搭建原型系统验证了所提出系统设计方法的可行性，以期促进先进人工智能技术与人机交互技术的融合发展，实现人机共融、安全协作。

关键词: 智能制造, 人机交互, 增强现实, 强化学习, 安全策略

Abstract: In modern manufacturing, the interaction and symbiosis between humans and the industrial robot are one of the foci of smart manufacturing. During human-robot interaction(HRI), the potential risk of any injury to workers caused by industrial robots is very critical and should be well-addressed to ensure manufacturing safety. However, in the dynamic and uncertain manufacturing environment, the current HRI safety is still based on the robot's perception of the environment to achieve collision avoidance, lacking adaptable decision-makings under mutual cognition. Therefore, to enhance the cognition of human operators in the working environment and improve the robot's collision avoidance and adaptive motion planning capabilities, this work designs and further implements a mutual cognitive HRI safety system based on augmented reality (AR) in a wearable manner. In the proposed system, the wearable AR device serves as the bridging interface to realize the virtual-real registration of the robot, the virtual-physical mapping of the working environment of the HRI process, and to collect the information of human, robot, and working space. In addition to these, a hierarchical HRI safety strategy is introduced for real-time mutual cognitive assistance to both humans and robots, namely: 1) robot motion speed control and safety area visualization based on human-robot distance, 2) virtual-physical mapping for robot motion preview and collision detection, and 3) deep reinforcement learning-driven motion planning for collision avoidance strategies generation. Lastly, a prototype system is further developed to validate the feasibility and effectiveness of the proposed strategies. By leveraging advanced artificial intelligence and human-robot interaction technologies, it is envisioned this work can bring insightful safety protection mechanisms to better achieve symbiotic human-robot collaboration.

Key words: smart manufacturing, human-robot interaction, augmented reality, reinforcement learning, safety strategy

中图分类号:

TP242

郑湃, 李成熙, 殷悦, 张荣, 鲍劲松, 王柏村, 谢海波, 王力翚. 增强现实辅助的互认知人机安全交互系统[J]. 机械工程学报, 2023, 59(6): 173-184.

ZHENG Pai, LI Chengxi, YIN Yue, ZHANG Rong, BAO Jinsong, WANG Baicun, XIE Haibo, WANG Lihui. Augmented Reality-assisted Mutual Cognitive System for Human-Robot Interaction Safety Concerns[J]. Journal of Mechanical Engineering, 2023, 59(6): 173-184.

参考文献

[1] 王田苗,陶永. 我国工业机器人技术现状与产业化发展战略[J]. 机械工程学报,2014,50(9):1-13. WANG Tianmiao,TAO Yong. Research status and industrialization development strategy of chinese industrial robot[J]. Journal of Mechanical Engineering,2014,50(9):1-13.
[2] 陶飞,戚庆林. 面向服务的智能制造[J]. 机械工程学报,2018,54(16):11-23. TAO Fei,QI Qinglin. Service-oriented smart manufacturing[J]. Journal of Mechanical Engineering,2018,54(16):11-23.
[3] Li S,Wang R,Zheng P,et al. Towards proactive human-robot collaboration:A foreseeable cognitive manufacturing paradigm[J]. Journal of Manufacturing Systems,2021,60:547-552.
[4] Bi Z M,Lang S Y T,Wang L. Improved control and simulation models of a tricycle collaborative robot[J]. Journal of Intelligent Manufacturing,2008,19(6):715-722.
[5] Bi Z M,Wang L. Dynamic control model of a cobot with three omni-wheels[J]. Robotics and Computer-Integrated Manufacturing,2010,26(6):558-563.
[6] BREQUE M,DE N L,PETRIDIS A. Industry 5.0:towards a sustainable,human-centric and resilient European industry[R]. Luxembourg,LU:European Commission,Directorate-General for Research and Innovation,2021.
[7] "新一代人工智能引领下的智能制造研究"课题组. 中国智能制造发展战略研究[J]. 中国工程科学,2018,20(4):1-8. The Research Group for Research on Intelligent Manufacturing Development Strategy. Research on intelligent manufacturing development strategy in China[J]. Strategic Study of Chinese Academy of Engineering,2018,20(4):1-8.
[8] XIAO M. The collaborative robot market 2021-28:grounds for optimism after a turbulent two years[EB/OL]. (Jan.2021)[Oct.2021] https://www.interactanalysis.com/the-collaborative-robot-market-2021-28-grounds-for-optimism-after-a-turbulent-two-years/.
[9] KUMAR S,SAVUR C,SAHIN F. Survey of human-robot collaboration in industrial settings:awareness,intelligence,and compliance[J]. IEEE Transactions on Systems,Man,and Cybernetics:Systems,2021,51(1):280-297.
[10] RODRÍGUEZ-GUERRA D,SORROSAL G,CABANES I,et al. Human-robot interaction review:challenges and solutions for modern industrial environments[J]. IEEE Access,2021,9:108557-108578.
[11] VICENTINI F,ASKARPOUR M,ROSSI M G,et al. Safety assessment of collaborative robotics through automated formal verification[J]. IEEE Transactions on Robotics,2020,36(1):42-61.
[12] 王柏村,黄思翰,易兵,等. 面向智能制造的人因工程研究与发展[J]. 机械工程学报,2020,56(16):240-253. WANG Baicun,HUANG Sihan,YI Bing,et al. State-of-art of human factors/ergonomics in intelligent manufacturing[J]. Journal of Mechanical Engineering,2020,56(16):240-253.
[13] 王柏村,薛塬,延建林,等. 以人为本的智能制造:理念、技术与应用[J]. 中国工程科学,2020,22(4):139-146. WANG Baicun,XUE Yuan,YAN Jianlin,et al. Human-centered intelligent manufacturing:overview and perspectives[J]. Strategic Study of Chinese Academy of Engineering,2020,22(4):139-146.
[14] International Organization for Standardization. ISO 10218-2:2011,Robots and robotic devices-Safety requirements for industrial robots-Part 2:Robot systems and integration[S]. Geneva,Switzerland:ISO,2011.
[15] International Organization for Standardization. ISO/TS 15066:2016,Robots and robotic devices-Collaborative robots[S]. Geneva,Switzerland:ISO,2016.
[16] BAROROH D K,CHU C H,WANG L. Systematic literature review on augmented reality in smart manufacturing:collaboration between human and computational intelligence[J]. Journal of Manufacturing Systems,2020. https://doi.org/10.1016/j.jmsy.2020.10.017.
[17] GALLALA A,HICHRI B,PLAPPER P. Survey:the evolution of the usage of augmented reality in industry 4.0[C]//IOP Conference Series:Materials Science and Engineering. IOP Publishing,2019,521(1):012017.
[18] EGGER J,MASOOD T. Augmented reality in support of intelligent manufacturing-a systematic literature review[J]. Computers & Industrial Engineering,2020,140:106196.
[19] BAKER A L,PHILLIPS E K,ULLMAN D,et al. Toward an understanding of rrust repair in human-robot interaction:Current research and future directions[J]. ACM Transactions on Interactive Intelligent Systems,2018,8(4):1-30.
[20] TATIC D,TESIC B. The application of augmented reality technologies for the improvement of occupational safety in an industrial environment[J]. Computers in Industry,2017,85:1-10.
[21] LI S,ZHENG P,ZHENG L. An AR-assisted deep learning-based approach for automatic inspection of aviation connectors[J]. IEEE Transactions on Industrial Informatics,2020,17(3):1721-1731.
[22] SINGH B,KUMAR R,SINGH V P. Reinforcement learning in robotic applications:A comprehensive survey[J]. Artificial Intelligence Review,2021(55):945-990.
[23] 贾计东,张明路. 人机安全交互技术研究进展及发展趋势[J]. 机械工程学报,2020,56(3):16-30. JIA Jidong,ZHANG Minglu. Research progress and development trend of the safety of human-robot interaction technology[J]. Journal of Mechanical Engineering,2020,56(3):16-30.
[24] ZACHARAKI A,KOSTAVELIS I,GASTERATORS A,et al. Safety bounds in human robot interaction:A survey[J]. Safety Science,2020,127:104667.
[25] 赵京,张自强,郑强,等. 机器人安全性研究现状及发展趋势[J]. 北京航空航天大学学报,2018,44(7):347-358. ZHAO Jing,ZHANG Ziqiang,ZHENG Qiang,et al. Research status and development trend of robot safety[J]. Journal of Beijing University of Aeronautics and Astronautics,2018,44(7):347-358.
[26] IKUTA K,NOKOTA M,ISHII H. Safety evaluation method of human-care robot control[C]//MHS2000. Proceedings of 2000 International Symposium on Micromechatronics and Human Science (Cat. No. 00TH8530). IEEE,2000:119-127.
[27] BI Z M,LUO C,MIAO Z,et al. Safety assurance mechanisms of collaborative robotic systems in manufacturing[J]. Robotics and Computer-Integrated Manufacturing,2021,67:102022.
[28] ZARDYKHAN D,SVARNY P,HOFFMANN M,et al. Collision preventing phase-progress control for velocity adaptation in human-robot collaboration[C]//2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids). Toronto,Canada:IEEE,2020:266-273.
[29] WANG X V,WANG L. Safety strategy and framework for human-robot collaboration[M]. Advanced Human-Robot Collaboration in Manufacturing. Springer,Cham,2021:69-87.
[30] WANG X,WANG L. Safety strategy in the smart manufacturing system:A human robot collaboration case study[C]//International Manufacturing Science and Engineering Conference. American Society of Mechanical Engineers,2020,84263:V002T07A026. https://doi.org/10.1115/MSEC2020-8427.
[31] 禹鑫燚,王正安,吴加鑫,等. 满足不同交互任务的人机共融系统设计[J/OL]. 自动化学报:1-12[2021-10-27]. YU Xinyi,WANG Zhengan,WU Jiaxin,et al. System design for human-robot coexisting environment satisfying multiple interaction tasks[J/OL]. Acta Automatica Sinica:1-12[2021-10-27].
[32] WANG X V,WANG L. Augmented reality enabled human-robot collaboration[M]. Advanced Human-Robot Collaboration in Manufacturing. London:Springer,Cham,2021.
[33] OZTEMEL O,GURSEV S. Literature review of industry 4.0 and related technologies[J]. Journal of Intelligent Manufacturing,2020,31(1):127-182.
[34] ARBELÁEZ J C,VIGANÒ R,OSORIO-GÓMEZ G. Haptic augmented reality (HapticAR) for assembly guidance[J]. International Journal on Interactive Design and Manufacturing,2019,13(2):673-687
[35] GREEN S A,BILLINGHURST M,CHEN X Q,et al. Human-robot collaboration:a literature review and augmented reality approach in design[J]. International Journal of Advanced Robotic Systems. 2008,5(1):1-18.
[36] VOGEL C,WALTER C,ELKMANN N. Safeguarding and supporting future human-robot cooperative manufacturing processes by a projection-and camera- based technology[J]. Procedia Manufacturing,2017,11:39-46
[37] HIETANEN A,PIETERS R,LANZ M,et al. AR-based interaction for human-robot collaborative manufacturing[J]. Robotics and Computer-Integrated Manufacturing,2020,63:101891.
[38] PAPANASTASIOU S,KOUSI N,KARAGIANNIS P,et al. Towards seamless human robot collaboration:integrating multimodal interaction[J]. The International Journal of Advanced Manufacturing Technology,2019,105(3):3881-3897.
[39] SIEW C Y,ONG S K,NEE A Y C. A practical augmented reality-assisted maintenance system framework for adaptive user support[J]. Robot Computer Integrated Manufacturing,2019,59:115-129.
[40] JOST J,KIRKS T,GUPTA P,et al. Safe human-robot-interaction in highly flexible warehouses using augmented reality and heterogenous fleet management system[C]//2018 IEEE International Conference on Intelligence and Safety for Robotics,Shengyang,China:IEEE,2018:256-260.
[41] 李浩,马文锋,文笑雨,等. 一种基于数字孪生的人-机交互安全预警与控制方法:中国,CN111563446A[P]. 2020-08-21. LI Hao,MA Wenfeng,WEN Xiaoyu,et al. A digital twin-based human-robot interaction safety warning and control method:China,CN111563446A[P]. 2020-08-21.
[42] LIN N,ZHANG L,CHEN Y,et al. Reinforcement learning for robotic safe control with force sensing[C]//2019 WRC Symposium on Advanced Robotics and Automation (WRC SARA). Beijing,China:IEEE,2019:148-153.
[43] LIU Q,LIU Z,XIONG B,et al. Deep reinforcement learning-based safe interaction for industrial human-robot collaboration using intrinsic reward function[J]. Advanced Engineering Informatics,2021,49(12):101360.
[44] SANGIOVANNI B,RENDINIELLO A,INCREMONA G P,et al. Deep reinforcement learning for collision avoidance of robotic manipulators[C]//2018 European Control Conference (ECC). Limassol,Cyprus:IEEE,2018:2063-2068.
[45] SCHULMAN J,WOLSKI F,DHARIWAL P,et al. Proximal policy optimization algorithms[EB/OL]. (Aug.2017)[Oct.2021] https://arxiv.org/abs/1707.06347
[46] BOTTOU L. Stochastic gradient descent tricks[C]//Neural Networks:Tricks of the Trade. Berlin,Heidelberg:Springer,2012:421-436.
[47] SUMMALA H. Brake reaction times and driver behavior analysis[J]. Transportation Human Factors,2000,2(3):217-226.

增强现实辅助的互认知人机安全交互系统

Augmented Reality-assisted Mutual Cognitive System for Human-Robot Interaction Safety Concerns

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	张党, 赵永宣, 王振军, 张映锋. 数据-知识混合驱动的离散制造系统智能控制体系构架研究[J]. 机械工程学报, 2024, 60(6): 1-10,20.
[2]	王无印, 黄子钊, 庄子龙, 方怀瑾, 秦威. 基于深度强化学习的自动化码头堆场场桥调度方法[J]. 机械工程学报, 2024, 60(6): 44-57.
[3]	赵阔, 王皂琦, 潘臻信, 潘扬华, 张中飞, 屈挺. 大数据驱动的快消品终端拜访“云-边”联动决策与优化[J]. 机械工程学报, 2024, 60(6): 58-68.
[4]	狄子钧, 袁东风, 李东阳, 梁道君, 周晓天, 信苗苗, 曹凤, 雷腾飞. 基于多尺度-高效通道注意力网络的刀具故障诊断方法[J]. 机械工程学报, 2024, 60(6): 82-90.
[5]	王跃飞, 王超, 许于涛, 孙睿, 肖锴, 王凯林. 边-云协同下智能制造单元作业的数字孪生任务调度方法[J]. 机械工程学报, 2024, 60(6): 137-152.
[6]	苏建涛, 董绍华, 朱诗敏. 多目标混合流水车间机器故障重调度问题研究[J]. 机械工程学报, 2024, 60(4): 438-448.
[7]	郭景华, 李文昌, 王班, 王靖瑶. 基于深度强化学习的网联混合动力汽车队列控制[J]. 机械工程学报, 2024, 60(2): 262-271.
[8]	袁小庆, 吴涛, 原勋, 王文东. 基于GSO-RF意图识别算法的全身助力外骨骼控制方法研究[J]. 机械工程学报, 2024, 60(17): 91-101.
[9]	王耀南, 谢核, 邓晶丹, 毛建旭, 李文龙, 张辉. 智能制造测量机器人关键技术研究综述[J]. 机械工程学报, 2024, 60(16): 1-18.
[10]	王文波, 张映锋, 顾寄南, 张耿, 完严. 数据驱动的离散制造系统性能退化机理建模与预测控制方法研究[J]. 机械工程学报, 2024, 60(16): 400-411.
[11]	张海伦, 许庆, 高博麟, 王建强, 李克强. 网联环境下驾驶人响应机制——一种交叉路口接近行为定性效应分析与定量驾驶模式提取案例[J]. 机械工程学报, 2024, 60(10): 22-47.
[12]	李文礼, 张祎楠, 石晓辉, 王梦昕. 基于博弈论的右转无信号交叉口行人行为模拟[J]. 机械工程学报, 2024, 60(10): 86-101.
[13]	曾迪, 郑玲, 李以农, 杨显通. 自动驾驶奖励函数贝叶斯逆强化学习方法[J]. 机械工程学报, 2024, 60(10): 245-260.
[14]	杨硕, 李时珍, 赵中原, 黄小鹏, 黄岩军. 基于时序差分学习模型预测控制的一体化自动驾驶换道策略[J]. 机械工程学报, 2024, 60(10): 329-338.
[15]	谭征宇, 张瑞佛, 刘芝孜, 金奕, 贺刚. 智能网联汽车人机交互信任研究现状与展望[J]. 机械工程学报, 2024, 60(10): 366-383.