XR驱动的煤矿辅助作业人机混合决策：架构、关键技术及应用

doi:10.3901/JME.2025.15.297

摘要/Abstract

摘要： 随着需求牵引与政策推动，煤矿机器人的研发与应用空前活跃，当前已可在少数场景中自主作业，代替煤矿操作员完成部分工作。但在煤矿辅助作业中，环境与任务的开放性、复杂性决定了煤矿操作员的深度参与仍然必不可少。在人机共融协作的背景下，亟需探索如何充分利用人的智能与机器智能优化煤矿辅助作业的决策过程。针对这一问题，首先回顾了相关概念与工作，将煤矿辅助作业中的决策归结为人机混合决策问题。然后基于人在回路的人机混合增强智能范式提出了XR驱动的煤矿辅助作业人机混合决策架构，介绍该架构的组成部分与协同运行关系。之后对实现该架构所需的各项关键技术及相应解决方案进行探讨。最后以采煤机滚筒停机维护为案例构建模拟场景，在所提出的架构下进行应用测试，以证实其可行性与有效性。结果表明，所提出的架构将人机混合决策过程扩展到虚拟空间，能够通过异构信息流的巧妙转化与传递，在满足作业任务要求的前提下将煤矿操作员与辅助作业机器人的决策合理整合，形成虚实融合、人机共智的决策模式，具有在煤矿辅助作业以及其他领域类似场景中的应用潜力。

关键词: 煤矿辅助作业, 人机混合增强智能, 人机混合决策, 虚拟现实, 增强现实

Abstract: With the demand-driven and policy-promoted development, the research and application of coal mine robots have become unprecedentedly active. Currently, these robots can autonomously operate in a few scenarios, replacing coal mine operators in performing certain tasks. However, the openness and complexity of the environment and tasks in auxiliary operations necessitate the continued deep involvement of coal mine operators. In the context of human-robot integration and collaboration, it is essential to explore how to fully leverage human intelligence and machine intelligence to optimize the decision-making process in coal mine auxiliary operations. First, relevant concepts and works are reviewed, and decision-making in coal mine auxiliary operations is defined as a human-robot hybrid decision-making problem. Then, based on the human-machine hybrid augmented intelligence paradigm of human in the loop, an XR driven human-robot hybrid decision-making architecture for coal mine auxiliary operations is proposed, and the components and collaborative operation relationship of the architecture are introduced. After that, key technologies and corresponding solutions required to implement this architecture are discussed. Finally, a simulation scenario is constructed using the case of shutdown maintenance for a shearer drum, with application testing conducted under the proposed architecture to verify feasibility and effectiveness. Results indicate that the proposed architecture extends the human-robot hybrid decision-making process into virtual space, enabling a reasonable integration of decision-making between coal mine operators and auxiliary operation robots through the clever transformation and transmission of heterogeneous information flows, thus forming a decision-making model characterized by virtual-real fusion and human-robot co-intelligence, which holds potential for application in coal mine auxiliary operations and similar scenarios in other fields.

Key words: auxiliary operations in coal mines, human-machine hybrid augmented intelligence, human-robot hybrid decision-making, virtual reality, augmented reality

中图分类号:

TP249

刘曙光, 谢嘉成, 王学文, 秦浪. XR驱动的煤矿辅助作业人机混合决策：架构、关键技术及应用[J]. 机械工程学报, 2025, 61(15): 297-313.

LIU Shuguang, XIE Jiacheng, WANG Xuewen, QIN Lang. XR-driven Human-robot Hybrid Decision-making for Auxiliary Operations in Coal Mines: Architecture, Key Technologies, and Applications[J]. Journal of Mechanical Engineering, 2025, 61(15): 297-313.

参考文献

[1] 王双明,申艳军,宋世杰,等."双碳"目标下煤炭能源地位变化与绿色低碳开发[J].煤炭学报,2023,48(7):2599-2612.WANG Shuangming,SHEN Yanjun,SONG Shijie,et al.Change of coal energy status and green and low-carbon development under the"dual carbon"goal[J]. Journal of China Coal Society,2023,48(7):2599-2612.
[2] 王国法.煤矿智能化最新技术进展与问题探讨[J].煤炭科学技术,2022,50(1):1-27.WANG Guofa. New technological progress of coal mine intelligence and its problems[J]. Coal Science and Technology,2022,50(1):1-27.
[3] 袁亮,吴劲松,杨科.煤炭安全智能精准开采关键技术与应用[J].采矿与安全工程学报,2023,40(5):861-868.YUAN Liang,WU Jinsong,YANG Ke. Key technology and its application of coal safety intelligent precision mining[J]. Journal of Mining&Safety Engineering,2023,40(5):861-868.
[4] 工业和信息化部等十七部门.关于印发"机器人+"应用行动实施方案的通知[EB/OL].[2023-01-18]. https://www.gov.cn/zhengce/zhengceku/2023-01/19/conte nt_5738112.htm.Ministry of Industry and Information Technology et al.Notice on issuing the" Robotics+"application action implementation plan[EB/OL].[2023-01-18]. https://www.gov.cn/zhengce/zhengceku/2023-01/19/conte nt_5738112.htm.
[5] 国家矿山安监局等七部门.关于印发《关于深入推进矿山智能化建设促进矿山安全发展的指导意见》的通知[EB/OL].[2024-04-24]. https://www.gov.cn/zhengce/zhengceku/202404/content_6948041.htm.National Mine Safety Administration et al. Notice on Issuing the" Guiding Opinions on Deepening the Promotion of Intelligent Mine Construction to Promote Safe Mine Development"[EB/OL].[2024-04-24]. https://www.gov.cn/zhengce/zhengceku/202404/content_6948041.htm.
[6] 葛世荣,胡而已,李允旺.煤矿机器人技术新进展及新方向[J].煤炭学报,2023,48(1):54-73.GE Shirong,HU Eryi,LI Yunwang. New progress and direction of robot technology in coal mine[J]. Journal of China Coal Society,2023,48(1):54-73.
[7] 国家煤矿安监局.煤矿机器人重点研发目录[EB/OL].[2019-01-09]. https://www.chinamine-safety.gov.cn/zfxxgk/fdzdgknr/tzgg/201901/t20190109_349156.shtml.National Coal Mine Safety Administration. Key R&D catalogue for coal mine robots[EB/OL].[2019-01-09]. https://www.chinamine-safety.gov.cn/zfxxgk/fdzdgknr/tzg g/201901/t20190109_349156.shtml.
[8] 张守祥,张学亮,张磊,等.综采巡检机器人关键技术研究[J].煤炭科学技术,2022,50(1):247-255.ZHANG Shouxiang,ZHANG Xueliang,ZHANG Lei,et al. Research on key technology of patrol robot in fully-mechanized mining face[J]. Coal Science and Technology,2022,50(1):247-255.
[9] 郝明锐.矿井轮式物料运输机器人设计[J].煤炭科学技术,2022,50(4):270-276.HAO Mingrui. Design of mine wheeled material transport robot[J]. Coal Science and Technology,2022,50(4):270-276.
[10] 葛世荣,胡而已,裴文良.煤矿机器人体系及关键技术[J].煤炭学报,2020,45(1):455-463.GE Shirong,HU Eryi, PEI Wenliang. Classification system and key technology of coal mine robot[J]. Journal of China Coal Society,2020,45(1):455-463.
[11] 王国法,庞义辉,任怀伟,等.智慧矿山系统工程及关键技术研究与实践[J].煤炭学报,2024,49(1):181-202.WANG Guofa,PANG Yihui,REN Huaiwei,et al. System engineering and key technologies research and practice of smart mine[J]. Journal of China Coal Society,2024,49(1):181-202.
[12] HENTOUT A,AOUACHE M,MAOUDJ A,et al.Human-robot interaction in industrial collaborative robotics:A literature review of the decade 2008-2017[J].Advanced Robotics,2019,33(15-16):764-799.
[13] 杨赓,周慧颖,王柏村.数字孪生驱动的智能人机协作:理论、技术与应用[J].机械工程学报,2022,58(18):279-291.YANG Geng,ZHOU Huiying,WANG Baicun. Digital twin-driven smart human-machine collaboration:Theory,enabling technol-ogies and applications[J]. Journal of Mechanical Engineering,2022,58(18):279-291.
[14] ZHOU J,LI P,ZHOU Y,et al. Toward new-generation intelligent manufacturing[J]. Engineering,2018,4(1):11-20.
[15] LENG J,SHA W,WANG B,et al. Industry 5.0:Prospect and retrospect[J]. Journal of Manufacturing Systems,2022,65:279-295.
[16] ZHOU J,ZHOU Y,WANG B,et al. Human-cyberphysical systems (HCPSs) in the context of newgeneration intelligent manufacturing[J]. Engineering,2019,5(4):624-636.
[17] 王柏村,薛塬,延建林,等.以人为本的智能制造:理念、技术与应用[J].中国工程科学,2020,22(4):139-146.WANG Baicun, XUE Yuan, YAN Jianlin, et al.Human-centered intel-ligent manufacturing:Overview and perspectives[J]. Strategic Study of CAE,2020,22(4):139-146.
[18] 姚锡凡,马南峰,张存吉,等.以人为本的智能制造:演进与展望[J].机械工程学报,2022,58(18):2-15.YAO Xifan, MA Nanfeng, ZHANG Cunji, et al.Human-centric smart manufacturing:Evolution and outlook[J]. Journal of Mechanical Engineering,2022,58(18):2-15.
[19] ZHENG P,LI S,XIA L,et al. A visual reasoning-based approach for mutual-cognitive human-robot collaboration[J].CIRP Annals-Manufacturing Technology,2022,71(1):377-380.
[20] LI X,CAO Z,XU Y. Characteristics and trends of coal mine safety development[J]. Energy Sources,Part A:Recovery,Utilization,and Environmental Effects,47(1):2316-2334.
[21] XIE J,LIU S,WANG X. Framework for a closed-loop cooperative human cyber-physical system for the mining industry driven by VR and AR:MHCPS[J]. Computers&Industrial Engineering,2022,168:108050.
[22] 谢嘉成,房舒凯,王学文,等."人本智造与XR+"驱动的综采工作面人机协同智能化运行模式探索与实践[J].煤炭学报,2023,48(2):1099-1114.XIE Jiacheng,FANG Shukai,WANG Xuewen,et al.Exploration and practice of the human-machine collaborative intelligent operation mode of fully mechanized coal mining face driven by humanistic intelligent manufacturing and XR+technology[J]. Journal of China Coal Society,2023,48(2):1099-1114.
[23] 王学文,刘曙光,王雪松,等.面向多人-多机复杂协作任务的煤矿XR智能运维系统[J].煤炭学报,2024,49(4):2124-2140.WANG Xuewen,LIU Shuguang,WANG Xuesong,et al.Research on coal mine XR intelligent operation and maintenance system for complex collaborative tasks involving multiple humans and multiple robots[J]. Journal of China Coal Society,2024,49(4):2124-2140.
[24] YANG C,ZHU Y,CHEN Y. A review of human-machine cooperation in the robotics domain[J]. IEEE Transactions on Human-Machine Systems,2022,52(1):12-25.
[25] GUPTA A,SAVARESE S,GANGULI S,et al. Embodied intelligence via learning and evolution[J]. Nature Communications,2021,12(1):5721.
[26] NYGAARD T F,MARTIN C P,TORRESEN J,et al.Real-world embodied AI through a morphologically adaptive quadruped robot[J]. Nature Machine Intelligence,2021,3(5):410-419.
[27] KOIVISTO M, GRASSINI S. Best humans still outperform artificial intelligence in a creative divergent thinking task[J]. Scientific Reports,2023,13(1):13601.
[28] ZHENG N,LIU Z,REN P,et al. Hybrid-augmented intelligence:Collaboration and cognition[J]. Frontiers of Information Technology&Electronic Engineering,2017,18(2):153-179.
[29] 国务院.国务院关于印发新一代人工智能发展规划的通知[EB/OL].[2017-07-20]. https://www.gov.cn/zhengce/content/2017-07/20/content_5211996.htm.State Council. State Council issues the" Development Plan for the New Generation of Artificial Intelligence"[EB/OL].[2027-07-20]. https://www.gov.cn/zhengce/content/2017-07/20/content_5211996.htm.
[30] XUE J,HU B,LI L,et al. Human-machine augmented intelligence:Research and applications[J]. Frontiers of Information Technology&Electronic Engineering,2022,23(8):1139-1141.
[31] REN M,CHEN N,QIU H. Human-machine collaborative decision-making:An evolutionary roadmap based on cognitive intelligence[J]. International Journal of Social Robotics,2023,15(7):1101-1114.
[32] KASE S E,HUNG C P,KRAYZMAN T,et al. The future of collaborative human-artificial intelligence decisionmaking for mission planning[J]. Frontiers in Psychology,2022,13.
[33] 范士雄,郭剑波,马士聪,等.混合增强智能在电力系统中的应用研究[J].电网技术,2023,47(10):4081-4091.FAN Shixiong, GUO Jianbo, MA Shicong, et al.Application analysis and exploration of hybrid-augmented intelligence in power systems[J]. Power System Technology,2023,47(10):4081-4091.
[34] 李鹏,黄文琦,梁凌宇,等.人机混合增强决策智能在新型电力系统调控中的应用与展望[J].中国电机工程学报,2024,44(16):6347-6367.LI Peng, HUANG Wenqi, LIANG Lingyu, et al.Application and prospect of hybrid human-machine decision intelligence in the dispatch and control of next-era power systems[J]. Proceedings of the CSEE,2024,44(16):6347-6367.
[35] FAN S,GUO J,MA S,et al. Framework and key technologies of human-machine hybrid-augmented intelligence system for large-scale power grid dispatching and control[J]. CSEE Journal of Power and Energy Systems,2024,10(1):1-12.
[36] 杨俊儒,褚端峰,陆丽萍,等.智能汽车人机共享控制研究综述[J].机械工程学报,2022,58(18):31-55.YANG Junru,CHU Duanfeng,LU Liping,et al. Review on human-machine shared control of intelligent vehicles[J]. Journal of Mechanical Engineering,2022,58(18):31-55.
[37] NING H,YIN R,ULLAH A,et al. A survey on hybrid human-artificial intelligence for autonomous driving[J].IEEE Transactions on Intelligent Transportation Systems,2022,23(7):6011-6026.
[38] YUAN K,HUANG Y,YANG S,er al. Evolutionary decision-making and planning for autonomous driving:A hybrid augmented intelligence framework[J]. IEEE Transactions on Intelligent Transportation Systems,2024,25(7):7339-7351.
[39] CÁRDENAS-ROBLEDO LA,HERNÁNDEZ-URIBEÓ,RETA C,et al. Extended reality applications in industry4.0.-A systematic literature review[J]. Telematics and Informatics,2022,73:101863.
[40] TAO F,ZHANG H,LIU A,et al. Digital twin in industry:State-of-the-art[J]. IEEE Transactions on Industrial Informatics,2019,15(4):2405-2415.
[41] SUN X,ZHANG R,LIU S,et al. A digital twin-driven human-robot collaborative assembly-commissioning method for complex products[J]. The International Journal of Advanced Manufacturing Technology,2022,118(9):3389-3402.
[42] LI H,MA W,WANG H,et al. A framework and method for human-robot cooperative safe control based on digital twin[J]. Advanced Engineering Informatics,2022,53:101701.
[43] DALLEL M,HAVARD V,DUPUIS Y,et al. Digital twin of an industrial workstation:A novel method of an auto-labeled data generator using virtual reality for human action recognition in the context of human-robot collaboration[J]. Engineering Applications of Artificial Intelligence,2023,118:105655.
[44] 郑湃,李成熙,殷悦,等.增强现实辅助的互认知人机安全交互系统[J].机械工程学报,2023,59(6):173-184.ZHENG Pai,LI Chengxi,YIN Yue,et al. Augmented reality-assisted mutual cognitive system for human-robot interaction safety concerns[J]. Journal of Mechanical Engineering,2023,59(6):173-184.
[45] CHOI S H,PARK K B,ROH D H,et al. An integrated mixed reality system for safety-aware human-robot collaboration using deep learning and digital twin generation[J]. Robotics and Computer-Integrated Manufacturing,2022,73:102258.
[46] CHU C H,LIU Y L. Augmented reality user interface design and experimental evaluation for human-robot collaborative assembly[J]. Journal of Manufacturing Systems,2023,68:313-324.
[47] AIVALIOTIS S,LOTSARIS K,GKOURNELOS C,et al. An augmented reality software suite enabling seamless human robot interaction[J]. International Journal of Computer Integrated Manufacturing,2023,36(1):3-29.
[48] LIU C, ZHANG Z, TANG D, et al. A mixed perception-based human-robot collaborative maintenance approach driven by augmented reality and online deep reinforcement learning[J]. Robotics and Computer-Integrated Manufacturing,2023,83:102568.
[49] ESWARAN M,INKULU A K,TAMILARASAN K,et al. Optimal layout planning for human robot collaborative assembly systems and visualization through immersive technologies[J]. Expert Systems with Applications,2024,241:122465.
[50] BARATTA A,CIMINO A,LONGO F,et al. Digital twin for human-robot collaboration enhancement in manufacturing systems:Literature review and direction for future developments[J]. Computers&Industrial Engineering,2024,187:109764.
[51] LIU S,XIE J,WANG X. Qo E enhancement of the industrial metaverse based on mixed reality application optimization[J]. Displays,2023,79:102463.
[52] LIU S,XIE J,WANG X,et al. Mixed reality collaboration environment improves the efficiency of human-centered industrial system:A case study in the mining industry[J].Computers&Industrial Engineering,2023,180:109257.
[53] LIU Z,LIU Q,XU W,et al. Human-robot collaborative manufacturing using cooperative game:Framework and implementation[J]. Procedia CIRP,2018,72:87-92.