基于动态视觉范式与实时反馈校验的脑控机械臂

doi:10.3901/JME.2023.21.157

机械工程学报 ›› 2023, Vol. 59 ›› Issue (21): 157-166.doi: 10.3901/JME.2023.21.157

扫码分享

基于动态视觉范式与实时反馈校验的脑控机械臂

张德雨¹, 刘思宇², 张健¹, 闫天翼², 吴景龙¹, 明致远¹

1. 北京理工大学机电学院北京 100081;
2. 北京理工大学生命学院北京 100081

收稿日期:2022-12-25 修回日期:2023-05-13 出版日期:2023-11-05 发布日期:2024-01-15
通讯作者: 闫天翼(通信作者)，男，1981年出生，教授，博士研究生导师。主要研究方向为脑机智能与神经调控。E-mail：yantianyi@bit.edu.cn
作者简介:张德雨，男，1993年出生，博士研究生。主要研究方向为脑机接口与智能机器人系统。E-mail：824074989@qq.com
基金资助:
MOST2030大脑项目(2022ZD0208500)、国家自然科学基金(U20A2019161727807,82071912,12104049,82202291)、中央高校基本科研业务费专项(2021CX11011)、中国博士后科学基金会(2020TQ0040,2022M710388)和BIT研究与创新促进项目(2022YCXZ026)资助项目。

Brain Controlled Manipulator Based on Dynamic Visual Paradigm and Real-time Feedback

ZHANG Deyu¹, LIU Siyu², ZHANG Jian¹, YAN Tianyi², WU Jinglong¹, MING Zhiyuan¹

1. School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081;
2. School of Life Science, Beijing Institute of Technology, Beijing 100081

Received:2022-12-25 Revised:2023-05-13 Online:2023-11-05 Published:2024-01-15

摘要/Abstract

摘要： 提出了一种应用于脑控机械臂的动态脑控视觉范式以及实时反馈的脑控指令校验方法，并基于该范式及方法实现了一套脑控机械臂系统。相比于当前广泛采用的固定视觉刺激，使用者通过注视不同位置控制机械臂末端运动过程的控制实现方法，提出的视觉范式基于动态刺激标签输出面向控制对象的脑控指令。该指令结合机械臂的自主空间运动，实现了基于脑-机协同下的快速抓取。同时，针对面向运动过程脑机接口中脑特征识别准确率低，系统鲁棒性不足的问题，提出了一种实时反馈校验方法，基于该方法对系统使用者的意图进行校验，可有效提升脑控机械臂对人类意图识别的准确率。实验结果表明，研究提出的动态刺激及实时反馈校验方法，其脑控抓取的平均成功率为85%(基于运动过程的控制方法的成功率为47.9%)，单次抓取任务下的平均时间消耗为22.46 s (基于运动过程的控制方法为64.76 s)。在未来，提出的脑控机械臂框架可以以节点模块的形式整合入机器人操作系统，并应用于人-机体系协同下的脑机接口闭环控制。

关键词: 脑机接口, 动态视觉范式, 实时反馈, 脑控机械臂

Abstract: A dynamic brain control vision paradigm and a real-time feedback brain command verification method applied to the brain control robot arm are proposed, and a set of brain control robotic arm system is realized based on the proposed paradigm and method. Compared with the fixed stimulus position and sequential control paradigm of brain computer interfaces, the paradigm outputs object-oriented brain control commands based on the dynamic stimulus, and fast capture based on brain-machine cooperation is realized. On the other hand, in view of the low accuracy of brain feature recognition and insufficient robustness of the system in the process-oriented brain computer interface, this research proposes a real-time feedback verification method. Based on this method, the raw brain command can be verified via dynamic frequency adjustment, which can effectively improve the accuracy of the brain controlled robot arm in brain command recognition. Result shows the dynamic stimulation and real-time feedback verification method has an average success rate of 85% (47.9% for the control method based on motion process) for brain controlled grasping, and an average time consumption of 22.46 s (64.76 s for the control method based on motion process) for a single grasping task. In the future, the framework of the brain controlled robot arm proposed can be integrated into the robot operating system in the form of node modules, and applied to the closed-loop control of the brain-machine cooperative systems.

Key words: brain computer interfaces, dynamic visual paradigm, real-time feedback, brain controlled robotic arm

中图分类号:

TP249

张德雨, 刘思宇, 张健, 闫天翼, 吴景龙, 明致远. 基于动态视觉范式与实时反馈校验的脑控机械臂[J]. 机械工程学报, 2023, 59(21): 157-166.

ZHANG Deyu, LIU Siyu, ZHANG Jian, YAN Tianyi, WU Jinglong, MING Zhiyuan. Brain Controlled Manipulator Based on Dynamic Visual Paradigm and Real-time Feedback[J]. Journal of Mechanical Engineering, 2023, 59(21): 157-166.

参考文献

[1] CHAMOLA V, VINEET A, NAYYAR A, et al. Brain-computer interface-based humanoid control:A review[J]. Sensors, 2020, 20(13):3620.
[2] SI-MOHAMMED H, PETIT J. Towards BCI-based interfaces for augmented reality:Feasibility, design and evaluation[J]. IEEE Transactions on Visualization and Computer Graphics, 2018(99):1.
[3] RASHID M, SULAIMAN N, MAJEED A, et al. Current status, challenges, and possible solutions of EEG-based brain-computer interface:A comprehensive review[J]. Frontiers in Neurorobotics, 2020(3):14-25.
[4] ZHANG W, TAN C, SUN F, et al. A review of EEG-based brain-computer interface systems design[J]. Brain Science Advances, 2018, 4(2):156-167.
[5] CHEN X, WANG Y, GAO S, et al. Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain-computer interface[J]. Journal of Neural Engineering, 2015, 12(4):046008.
[6] LI Z, YUAN W, ZHAO S, et al. Brain-actuated control of dual-arm robot manipulation with relative motion[J]. IEEE Transactions on Cognitive and Developmental Systems, 2019, 11(1):51-62.
[7] ARZU G, AKIN H L. An SSVEP based BCI to control a humanoid robot by using portable EEG device[C]//35th Annual Intornational Conference of the IEEE Engineeringin Medicine and Biology society, Osaka, Japan, 2013:6905-6908.
[8] CHEN X, HUANG X, WANG Y, et al. Combination of augmented reality based brain-computer interface and computer vision for high-level control of a robotic arm[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(12):3140-3147.
[9] YANG C, WU H, LI Z, et al. Mind control of a robotic arm with visual fusion technology[J]. IEEE Transactions on Industrial Informatics, 2018, 14(9):3822-3830.
[10] REBSAMEN B, GUAN C, ZHANG H, et al. A Brain controlled wheelchair to navigate in familiar environments[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2010, 18(6):590-598.
[11] WU Y, LI M, WANG J. Toward a hybrid brain-computer interface based on repetitive visual stimuli with missing events[J]. Journal of NeuroEngineering and Rehabilitation, 2016, 13(1):1-12.
[12] THOMPSON B. Canonical correlation analysis[M]. American Cancer Society, John Wiley & Sons, Ltd, United States, 2005.
[13] INGEL A, KUZOVKIN I, VICENTE R. Direct information transfer rate optimisation for SSVEP-based BCI[J]. Journal of Neural Engineering, 2019, 16(1):228-240.
[14] 王帅,王子腾,郑德智. 一种基于SSVEP脑机接口的仪器仪表控制系统:中国, 110850795A[P]. 2020-02-28. WANG Shuai, WANG Ziteng, ZHENG Dezhi. An instrument control system based on SSVEP brain computer interface:China, 110850795A[P]. 2020-02-28.
[15] 任士鑫,王卫群,侯增广,等. 基于改进共空间模式与视觉反馈的闭环脑机接口[J]. 机械工程学报, 2019, 55(11):28-35. REN Shixin, WANG Weiqun, HOU Zengguang, et al Closed loop brain computer interface based on improved common space mode and visual feedback[J] Journal of Mechanical Engineering, 2019, 55(11):28-35.
[16] MULLER-PUTZ G R, PFURTSCHELLER G. Control of an electrical prosthesis with an SSVEP-Based BCI[J]. IEEE Transactions on Biomedical Engineering, 2007, 55(1):361-364.
[17] FANG B. Brain-computer interface integrated with augmented reality for human-robot interaction[J]. IEEE Transactions on Cognitive and Developmental Systems, 2022(7):1.
[18] 程杨,潘尚峰. 一种多自由度康复外骨骼机械臂的虚拟分解控制[J]. 机械工程学报, 2022, 58(9):21-30. CHENG Yang, PAN Shangfeng. Virtual decomposition control of a multi DOF rehabilitation exoskeleton manipulator[J]. Journal of Mechanical Engineering, 2022, 58(9):21-30.

基于动态视觉范式与实时反馈校验的脑控机械臂

Brain Controlled Manipulator Based on Dynamic Visual Paradigm and Real-time Feedback

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 2

编辑推荐

Metrics

本文评价

[1]	梁野, 裘乐淼, 刘晓健, 张树有, 谭建荣. 基于设计情境与脑机反馈融合的产品设计知识需求感知技术[J]. 机械工程学报, 2020, 56(7): 151-163.
[2]	任士鑫, 王卫群, 侯增广, 陈霸东, 石伟国, 王佳星, 梁旭. 基于改进共空间模式与视觉反馈的闭环脑机接口[J]. 机械工程学报, 2019, 55(11): 28-35.