Research Progress of Sensory Feedback for Intelligent Upper-limb Prosthesis

doi:10.3901/JME.2023.05.001

Abstract

Abstract: In recent years, the bidirectional bio-machine interface with the ability of neural control and sensory feedback has become the development trends of intelligent upper limb prosthesis. Most of the research focuses on the mechanism, sensor design and neural control of upper limb prosthesis, while less research on sensory feedback. The lack of reliable sensory feedback reduces the operational performance and limits the practical application of prosthetics. Firstly, the development status of bidirectional bio-machine interface of intelligent prosthesis is briefly summarized. Sensory feedback methods based on transcutaneous electrical nerve stimulation, vibration stimulation and pressure stimulation are introduced in detail. Sensory feedback strategies including sensory substitution, modality matched feedback and somatotopic matched feedback are also discussed. Based on the analysis, the naturalness of upper limb prosthetic sensory feedback and the multimodality of interactive information are prospected.

Key words: intelligent prosthesis, bio-machine interface, sensory feedback, electrical stimulation

CLC Number:

TP241

HU Yawen, JIANG Li, YANG Bin. Research Progress of Sensory Feedback for Intelligent Upper-limb Prosthesis[J]. Journal of Mechanical Engineering, 2023, 59(5): 1-10.

References

[1] LI C,LI G,JIANG G,et al. Surface EMG data aggregation processing for intelligent prosthetic action recognition[J]. Neural Computing and Applications,2018,32(22):16795-16806.
[2] BENSMAIA S,TYLER D,MICERA S. Restoration of sensory information via bionic hands[J]. Nature Biomedical Engineering,2020:1-13.
[3] OSBORN L E,DING K,HAYS M A,et al. Sensory stimulation enhances phantom limb perception and movement decoding[J]. Journal of Neural Engineering,2020,17(5):056006.
[4] JABBAN L,DUPAN S,ZHANG D,et al. Sensory feedback for upper-limb prostheses:Opportunities and Barriers[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering,2022,30:738-747.
[5] BECERRA-FAJARDO L,KROB M,MINGUILLON J,et al. Floating EMG sensors and stimulators wirelessly powered and operated by volume conduction for networked neuroprosthetics[J]. Journal of NeuroEngineering and Rehabilitation,BioMed Central,2022,19(1):1-19.
[6] CHRISTIE B,OSBORN L E,MCMULLEN D,et al. Perceived timing of cutaneous vibration and intracortical microstimulation of human somatosensory cortex[J]. Brain Stimulation,2022,15(3):881-888.
[7] HAO M,CHOU C,ZHANG J,et al. Restoring finger-specific sensory feedback for transradial amputees via non-invasive evoked tactile sensation[J]. IEEE Open Journal of Engineering in Medicine and Biology,2020,1:98-107.
[8] VARGAS L,HUANG H,ZHU Y,et al. Object recognition via evoked sensory feedback during control of a prosthetic hand[J]. IEEE Robotics and Automation Letters,2022,7(1):207-214.
[9] WANG H,CHAI G,SHENG X,et al. A programmable,multichannel,miniature stimulator for electrotactile feedback of neural hand prostheses[C]//IEEE/EMBS Conference on Neural Engineering,2021:1026-1029.
[10] GRAUPE D,CLINE W. Functional separation of EMG signal via ARMA identification methods for prosthesis control purposes[J]. IEEE Transactions on Systems,1975,5(2):252-259.
[11] CONNOLLY C. Prosthetic hands from touch bionics[J]. Industrial Robot,2008,35(4):290-293.
[12] MULVEY M,FAWKNER H,RADFORD H,et al. Perceptual embodiment of prosthetic limbs by transcutaneous electrical nerve stimulation[J]. Neuromodulation:Technology at the Neural Interface,2012,15(1):42-47.
[13] KATO R,YOKOI H,HERNANDEZ A,et al. Mutual adaptation among man and machine by using f-MRI analysis[J]. Robotics and Autonomous Systems,2009,57(2):161-166.
[14] PATEL G,DOSEN S,CASTELLINI C,et al. Multichannel electrotactile feedback for simultaneous and proportional myoelectric control[J]. Journal of Neural Engineering,2016,13(5):056015.
[15] STRBAC M,ISAKOVIC M,BELIC M,et al. Short- and long-term learning of feedforward control of a myoelectric prosthesis with sensory feedback by amputees[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering,2017,25(11):2133-2145.
[16] OSBORN L E,DRAGOMIR A,BETTHAUSER J,et al. Prosthesis with neuromorphic multilayered e-dermis perceives touch and pain[J]. Science Robotics,2018,3(19).
[17] 宋爱国,吴常铖. 具有自主学习能力和力触觉感知反馈能力的智能肌电假手[J]. 科技纵览,2016,(11):69-71. SONG Aiguo,WU Changcheng. Intelligent EMG artificial hand with autonomous learning ability and force tactile perception feedback ability[J]. IEEE Spectrum,2016,(11):69-71.
[18] 胡旭晖,石珂,祝佳航,等. 基于力触觉感知与电刺激反馈的灵巧假手[J]. 机械工程学报,2019,55(11):10-18. HU Xuhui,SHI Ke,ZHU Jiahang,et al. Dexterous prosthetic hand with force perception and electrical stimulation[J]. Journal of Mechanical Engineering,2019,55(11):10-18.
[19] YIMENG G,SHUAN Y,KAIHUA Z,et al. Mediating different-diameter Aβ nerve fibers using a biomimetic 3D TENS computational model[J]. Journal of Neuroscience Methods,Elsevier,2020,346:108891.
[20] 姜力,杨斌,黄琦,等. 智能假肢手的生机电集成[J]. 机器人,2017,39(4):387-394. JIANG Li,YANG Bin,HUANG Qi,et al. Biomechatronic integration of intelligent prosthetic hand[J]. Robot,2017,39(4):387-394.
[21] MARASCO P,HEBERT J,SENSINGER J,et al. Illusory movement perception improves motor control for prosthetic hands[J]. Science Translational Medicine,2018,10(432).
[22] PAGE D M,GEORGE J A,KLUGER D T,et al. Motor control and sensory feedback enhance prosthesis embodiment and reduce phantom pain after long-term hand amputation[J]. Frontiers in Human Neuroscience,2018,12.
[23] MENDEZ V,IBERITE F,SHOKUR S,et al. Current solutions and future trends for robotic prosthetic hands[J]. Annual Review of Control Robotics and Autonomous Systems,2021,4(1):595-627.
[24] ISKAROUS M,THAKOR N. E-Skins:biomimetic sensing and encoding for upper limb prostheses[J]. Proceedings of the IEEE,2019,107(10):2052-2064.
[25] OSBORN L,NGUYEN H,BETTHAUSER J,et al. Biologically inspired multi-layered synthetic skin for tactile feedback in prosthetic limbs[C]//International Conference of the IEEE Engineering in Medicine & Biology Society. IEEE,2016:4622-4625.
[26] SHEN Z,ZHU X,MAJIDI C,et al. Cutaneous ionogel mechanoreceptors for soft machines,physiological sensing,and amputee prostheses[J]. Advanced Materials,2021,33(38).
[27] SVENSSON P,WIJK U,BJRKMAN A,et al. A review of invasive and non-invasive sensory feedback in upper limb prostheses[J]. Expert Review of Medical Devices,2017,14(6):439-447.
[28] KACZMAREK K,WEBSTER J,BACH-Y-RITA P,et al. Electrotactile and vibrotactile displays for sensory substitution systems[J]. IEEE Transactions on Biomedical Engineering,1991,38(1):1-16.
[29] ARAKERI T,HASSE B,FUGLEVAND A. Object discrimination using electrotactile feedback[J]. Journal of Neural Engineering,2018,15(4):12-25.
[30] KAJIMOTO H,KAWAKAMI N,MAEDA T,et al. Electro-tactile display with tactile primary color approach[C]//Proceedings of International Conference on Intelligent Robots and Systems. Sendai,Japan. 2004.
[31] HIRATA T,NAKAMURA T,KATO R,et al. Development of mobile controller for EMG prosthetic hand with tactile feedback[C]//2011 IEEE/ASME International Conference on Advanced Intelligent Mechatronics:Institute of Electrical and Electronics Engineers. IEEE,2011:110-115.
[32] PATRICIU A, YOSHIDA K, STRUIJK J, et al. Current density imaging and electrically induced skin burns under surface electrodes[J]. IEEE Transactions on Biomedical Engineering,2005,52(12):2024-2031.
[33] AKHTAR A,SOMBECK J,BOYCE B,et al. Controlling sensation intensity for electrotactile stimulation in human-machine interfaces[J]. Science Robotics,2018,3(17):eaap9770.
[34] JIANG L,HUANG Q,ZHAO J,et al. Noise cancellation for electrotactile sensory feedback of myoelectric forearm prostheses[C]//2014 IEEE International Conference on Information and Automation (ICIA). IEEE,2014.
[35] CONZELMAN J,ELLIS H,O'BRIEN C. Prosthetic device sensory attachment:US,1952265872[P]. 1953-10-27.
[36] MANN RW. Kinesthetic sensing for the EMG controlled "Boston Arm"[J]. IEEE Transactions on Man-Machine Systems,1970,11(1):110-115.
[37] PYLATIUK C,KARGOV A,SCHULZ S. Design and evaluation of a low-cost force feedback system for myoelectric prosthetic hands[J]. JPO Journal of Prosthetics and Orthotics,2006,18(2):57-61.
[38] 李楠,刘波,霍宏,等. 基于肌力信号与电刺激感觉反馈的多自由度机械手人机交互控制[J]. 机器人,2015,37(6):718-724. LI Nan,LIU Bo,HUO Hong,et al. Human-machine interaction control based on force myograph and electrical stimulation sensory feedback for multi-dof robotic hand[J]. Robot,2015,37(6):718-724.
[39] ANTFOLK C,BJRKMAN A,FRANK S O,et al. Sensory feedback from a prosthetic hand based on air-mediated pressure from the hand to the forearm skin[J]. Journal of rehabilitation medicine:official journal of the UEMS European Board of Physical and Rehabilitation Medicine,2012,44(8):702-707.
[40] MOAED A. ABD,JOSEPH INGICCO,DOUGLAS T. H,et al. Multichannel haptic feedback unlocks prosthetic hand dexterity[J]. Scientific Reports,Nature Publishing Group,2022,12(1):1-17.
[41] CASINI S,MORVIDONI M,BIANCHI M,et al. Design and realization of the CUFF-clenching upper-limb force feedback wearable device for distributed mechano-tactile stimulation of normal and tangential skin forces[C]//IEEE/RSJ International Conference on Intelligent Robots & Systems. IEEE,2015.
[42] PANARESE A,EDIN B,VECCHI F,et al. Humans can integrate force feedback to toes in their sensorimotor control of a robotic hand[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2009, 17(6):560-567.
[43] PYLATIUK C,KARGOV A,SCHULZ S. Design and evaluation of a low-cost force feedback system for myoelectric prosthetic hands[J]. Jpo Journal of Prosthetics & Orthotics,2006,18(2):57-61.
[44] PATEL G K,DOSEN S,CASTELLINI C,et al. Multichannel electrotactile feedback for simultaneous and proportional myoelectric control[J]. Journal of neural engineering,2016,13(5):056015.1-056015.13.
[45] SAUNDERS I,VIJAYAKUMAR S. The role of feed-forward and feedback processes for closed-loop prosthesis control[J]. Journal of Neuroengineering & Rehabilitation,2011,8(1):60-60.
[46] AJOUDANI A,GODFREY S B,BIANCHI M,et al. Exploring teleimpedance and tactile feedback for intuitive control of the Pisa/IIT SoftHand[J]. IEEE Transactions on Haptics,2014,7(2):203-215.
[47] JIMENEZ M,FISHEL J. Evaluation of force,vibration and thermal tactile feedback in prosthetic limbs[C]//IEEE Haptics Symposium,2014:437-441.
[48] STEPP C,MATSUOKA Y. Relative to direct haptic feedback,remote vibrotactile feedback improves but slows object manipulation[C]//IEEE Engineering in Medicine and Biology,2010:2089-2092.
[49] RASPOPOVIC S,VALLE G,PETRINI F. Sensory feedback for limb prostheses in amputees[J]. Nature Materials,2021,20(7):925-939.
[50] GEORGE J,KLUGER D,DAVIS T,et al. Biomimetic sensory feedback through peripheral nerve stimulation improves dexterous use of a bionic hand[J]. Science Robotics,2019,4(32):2352-2363.
[51] BJORKMAN A,WIJK U,ANTFOLK C,et al. Sensory qualities of the phantom hand map in the residual forearm of amputees[J]. Journal of Rehabilitation Medicine,2016,48(4):365-370.
[52] DIMANTE D,LOGINA I,SINISI M,et al. Sensory feedback in upper limb prostheses[J]. Proceedings of the Latvian Academy of Sciences,2020,74(5):308-317.
[53] CHAI G,SUI X,LI S,et al. Characterization of evoked tactile sensation in forearm amputees with transcutaneous electrical nerve stimulation[J]. Journal of Neural Engineering,2015,12(6):066002.
[54] MORRIS M,DY C,BOYER M,et al. Targeted muscle reinnervation and the volar forearm filet flap for forequarter amputation:description of operative technique[J]. Journal of Hand Surgery Global Online,2020,2(5):306-311.
[55] D'ANNA E,VALLE G,MAZZONI A,et al. A closed-loop hand prosthesis with simultaneous intraneural tactile and position feedback[J]. Science Robotics,2019,4(27).
[56] WODLINGER B, DOWNEY J, TYLER-KABARA E, et al. Ten-dimensional anthropomorphic arm control in a human brain-machine interface:difficulties, solutions, and limitations[J]. Journal of Neural Engineering, 2015, 12(1).
[57] FRANCESCA C,LISA C,RINALDO S,et al. Literature review on needs of upper limb prosthesis users[J]. Frontiers in Neuroscience,2016,10:209.
[58] SCHOFIELD J,EVANS K,CAREY J,et al. Applications of sensory feedback in motorized upper extremity prosthesis:a review[J]. Expert Review of Medical Devices,2014,11(5):499-511.
[59] ANTFOLK C,D' ALONZO M,CONTROZZI M,et al. Artificial redirection of sensation from prosthetic fingers to the phantom hand map on transradial amputees:vibrotactile versus mechanotactile sensory feedback[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering,2013,21(1):112-120.
[60] VALLE G,PETRINI F,STRAUSS I,et al. Comparison of linear frequency and amplitude modulation for intraneural sensory feedback in bidirectional hand prostheses[J]. Scientific Reports,Nature Publishing Group,2018,8(1):16666.
[61] GENG B,DONG J,JENSEN W,et al. Psychophysical svaluation of subdermal electrical stimulation in relation to prosthesis sensory feedback[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering,2018,26(3):709-715.
[62] GUÉMANN M,HALGAND C,BASTIER A,et al. Sensory substitution of elbow proprioception to improve myoelectric control of upper limb prosthesis:experiment on healthy subjects and amputees[J]. Journal of NeuroEngineering and Rehabilitation,2022,19(1):59.
[63] BLUSTEIN D,WILSON A,SENSINGER J. Assessing the quality of supplementary sensory feedback using the crossmodal congruency task[J]. Scientific Reports,2018,8(1):6203.
[64] SU S,CHAI G,MENG J,et al. Towards optimizing the non-invasive sensory feedback interfaces in a neural prosthetic control[J]. Journal of Neural Engineering,2022,19(1):16-28.
[65] WANG Y,FANG P,TANG X,et al. Effective evaluation of finger sensation evoking by non-invasive stimulation for sensory function recovery in transradial amputees[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering,2022,30:519-528.