Brain Controlled Manipulator Based on Dynamic Visual Paradigm and Real-time Feedback

doi:10.3901/JME.2023.21.157

Abstract

Abstract: A dynamic brain control vision paradigm and a real-time feedback brain command verification method applied to the brain control robot arm are proposed, and a set of brain control robotic arm system is realized based on the proposed paradigm and method. Compared with the fixed stimulus position and sequential control paradigm of brain computer interfaces, the paradigm outputs object-oriented brain control commands based on the dynamic stimulus, and fast capture based on brain-machine cooperation is realized. On the other hand, in view of the low accuracy of brain feature recognition and insufficient robustness of the system in the process-oriented brain computer interface, this research proposes a real-time feedback verification method. Based on this method, the raw brain command can be verified via dynamic frequency adjustment, which can effectively improve the accuracy of the brain controlled robot arm in brain command recognition. Result shows the dynamic stimulation and real-time feedback verification method has an average success rate of 85% (47.9% for the control method based on motion process) for brain controlled grasping, and an average time consumption of 22.46 s (64.76 s for the control method based on motion process) for a single grasping task. In the future, the framework of the brain controlled robot arm proposed can be integrated into the robot operating system in the form of node modules, and applied to the closed-loop control of the brain-machine cooperative systems.

Key words: brain computer interfaces, dynamic visual paradigm, real-time feedback, brain controlled robotic arm

CLC Number:

TP249

ZHANG Deyu, LIU Siyu, ZHANG Jian, YAN Tianyi, WU Jinglong, MING Zhiyuan. Brain Controlled Manipulator Based on Dynamic Visual Paradigm and Real-time Feedback[J]. Journal of Mechanical Engineering, 2023, 59(21): 157-166.

References

[1] CHAMOLA V, VINEET A, NAYYAR A, et al. Brain-computer interface-based humanoid control:A review[J]. Sensors, 2020, 20(13):3620.
[2] SI-MOHAMMED H, PETIT J. Towards BCI-based interfaces for augmented reality:Feasibility, design and evaluation[J]. IEEE Transactions on Visualization and Computer Graphics, 2018(99):1.
[3] RASHID M, SULAIMAN N, MAJEED A, et al. Current status, challenges, and possible solutions of EEG-based brain-computer interface:A comprehensive review[J]. Frontiers in Neurorobotics, 2020(3):14-25.
[4] ZHANG W, TAN C, SUN F, et al. A review of EEG-based brain-computer interface systems design[J]. Brain Science Advances, 2018, 4(2):156-167.
[5] CHEN X, WANG Y, GAO S, et al. Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain-computer interface[J]. Journal of Neural Engineering, 2015, 12(4):046008.
[6] LI Z, YUAN W, ZHAO S, et al. Brain-actuated control of dual-arm robot manipulation with relative motion[J]. IEEE Transactions on Cognitive and Developmental Systems, 2019, 11(1):51-62.
[7] ARZU G, AKIN H L. An SSVEP based BCI to control a humanoid robot by using portable EEG device[C]//35th Annual Intornational Conference of the IEEE Engineeringin Medicine and Biology society, Osaka, Japan, 2013:6905-6908.
[8] CHEN X, HUANG X, WANG Y, et al. Combination of augmented reality based brain-computer interface and computer vision for high-level control of a robotic arm[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(12):3140-3147.
[9] YANG C, WU H, LI Z, et al. Mind control of a robotic arm with visual fusion technology[J]. IEEE Transactions on Industrial Informatics, 2018, 14(9):3822-3830.
[10] REBSAMEN B, GUAN C, ZHANG H, et al. A Brain controlled wheelchair to navigate in familiar environments[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2010, 18(6):590-598.
[11] WU Y, LI M, WANG J. Toward a hybrid brain-computer interface based on repetitive visual stimuli with missing events[J]. Journal of NeuroEngineering and Rehabilitation, 2016, 13(1):1-12.
[12] THOMPSON B. Canonical correlation analysis[M]. American Cancer Society, John Wiley & Sons, Ltd, United States, 2005.
[13] INGEL A, KUZOVKIN I, VICENTE R. Direct information transfer rate optimisation for SSVEP-based BCI[J]. Journal of Neural Engineering, 2019, 16(1):228-240.
[14] 王帅,王子腾,郑德智. 一种基于SSVEP脑机接口的仪器仪表控制系统:中国, 110850795A[P]. 2020-02-28. WANG Shuai, WANG Ziteng, ZHENG Dezhi. An instrument control system based on SSVEP brain computer interface:China, 110850795A[P]. 2020-02-28.
[15] 任士鑫,王卫群,侯增广,等. 基于改进共空间模式与视觉反馈的闭环脑机接口[J]. 机械工程学报, 2019, 55(11):28-35. REN Shixin, WANG Weiqun, HOU Zengguang, et al Closed loop brain computer interface based on improved common space mode and visual feedback[J] Journal of Mechanical Engineering, 2019, 55(11):28-35.
[16] MULLER-PUTZ G R, PFURTSCHELLER G. Control of an electrical prosthesis with an SSVEP-Based BCI[J]. IEEE Transactions on Biomedical Engineering, 2007, 55(1):361-364.
[17] FANG B. Brain-computer interface integrated with augmented reality for human-robot interaction[J]. IEEE Transactions on Cognitive and Developmental Systems, 2022(7):1.
[18] 程杨,潘尚峰. 一种多自由度康复外骨骼机械臂的虚拟分解控制[J]. 机械工程学报, 2022, 58(9):21-30. CHENG Yang, PAN Shangfeng. Virtual decomposition control of a multi DOF rehabilitation exoskeleton manipulator[J]. Journal of Mechanical Engineering, 2022, 58(9):21-30.