面向动作识别的旋转投影变换关节特征骨架描述

doi:10.3901/JME.2021.05.019

机械工程学报 ›› 2021, Vol. 57 ›› Issue (5): 19-30.doi: 10.3901/JME.2021.05.019

扫码分享

面向动作识别的旋转投影变换关节特征骨架描述

巫晓康, 赵欢, 唐敏杰, 丁汉

华中科技大学数字制造装备与技术国家重点实验室武汉 430074

收稿日期:2020-03-15 修回日期:2020-08-20 出版日期:2021-03-05 发布日期:2021-04-28
通讯作者: 赵欢(通信作者),男,1983年出生,教授。主要研究方向为机器人智能化加工装备与技术。E-mail:huanzhao@hust.edu.cn
作者简介:巫晓康,男,1996年出生。主要研究方向为机器人视觉与人机协作。E-mail:xkwu@hust.edu.cn;唐敏杰,男,1995年出生,硕士研究生。主要研究方向为智能传感技术与机器人视觉。E-mail:tamgminjie@hust.edu.cn;丁汉,男,1963年出生,教授,博士研究生导师。主要研究方向为数字化制造与机器人技术。E-mail:dinghan@hust.edu.cn
基金资助:
国家自然科学基金（52090054）和湖北省自然科学基金（2020CFA077）资助项目。

Rotation and Projection Transformed Joint Feature Descriptors of Human Skeletal Action Recognition

WU Xiaokang, ZHAO Huan, TANG Minjie, DING Han

State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan 430074

Received:2020-03-15 Revised:2020-08-20 Online:2021-03-05 Published:2021-04-28

摘要/Abstract

摘要： 在机器人领域中，基于人类骨架关节特征描述的动作识别方法被广泛地应用于人机协作任务，以帮助机器人理解人类的行为。对动作序列中的时间信息和空间信息挖掘是其中的一个重要问题，不充分的信息提取往往会导致算法鉴别力低。鉴于此，从骨架旋转投影出发，提出了一种能够充分挖掘动作序列中时空信息的高鉴别力人类骨架关节特征描述方法，即骨架旋转投影描述子（Rotational and projective skeleton signature，RPSS）。建立以人为中心的正交坐标系；从训练序列中学习躯干尺寸并进行替换；将骨架分别绕坐标轴旋转一系列固定间隔的角度，并向某一平面进行投影以获取足量的空间信息，将多次投影得到的子特征进行串联后依据帧顺序并联得到序列的特征；利用方向梯度直方图对特征矩阵进行提取以生成最终的基于关节特征且蕴含丰富时空信息的描述子RPSS。所提出的RPSS描述子在多个公开的经典数据集上进行了测试，并与基于关节描述的经典和与学习相结合的先进骨架识别方法进行对比，结果表明所提出的方法在鉴别力上具有较大的优势。添加不同程度噪声与计算动作序列平均处理时长的试验亦说明该方法具有较强的鲁棒性和实时性。

关键词: 人类骨架, 特征提取, 旋转与投影, 动作识别

Abstract: In robotics, human skeletal action recognition with joint-based representations are widely used in human-robot collaboration. Mining temporal and spatial information in action sequence is a challenging problem. Insufficient information extraction tends to bring the algorithm less discriminability. In view of this, a high-performance joint-based descriptor called rotational and projective skeleton signature (RPSS) is proposed. First of all, build person-centric coordinate system; then learn average limbs (skeleton segments) lengths from the training set; Secondly, rotate the skeleton by a set of angles, meanwhile project the rotated skeletons on a certain plane to get the sequence-features; Finally, use histogram of oriented gradient (HOG) to exact spatio-tempral information to generate a final RPSS description. The RPSS descriptors were tested on a number of public datasets and compared with classical and learning based description algorithms. The results show that it has decent discriminative power, and the experiments adding different degrees of noises and recording average calculation time show it also has strong robustness and real-time performance.

Key words: skeleton, feature extraction, rotation & projection, human action recognition

中图分类号:

TP273

巫晓康, 赵欢, 唐敏杰, 丁汉. 面向动作识别的旋转投影变换关节特征骨架描述[J]. 机械工程学报, 2021, 57(5): 19-30.

WU Xiaokang, ZHAO Huan, TANG Minjie, DING Han. Rotation and Projection Transformed Joint Feature Descriptors of Human Skeletal Action Recognition[J]. Journal of Mechanical Engineering, 2021, 57(5): 19-30.

参考文献

[1] AGGARWAL J K,RYOO M S. Human activity analysis:A review[J]. ACM Computing Surveys (CSUR),2011,43(3):16.
[2] VEMULAPALLI R,ARRATE F,CHELLAPPA R. Human action recognition by representing 3D skeletons as points in a lie group[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Ohio:IEEE Computer Society,2014:588-595.
[3] ROITBERG A,SOMANI N,PERZYLO A,et al. Multimodal human activity recognition for industrial manufacturing processes in robotic workcells[C]//Proceedings of the 2015 ACM on International Conference on Multimodal Interaction,Seattle:ACM,2015:259-266.
[4] LIU C,TOMIZUKA M. Algorithmic safety measures for intelligent industrial co-robots[C]//2016 IEEE International Conference on Robotics and Automation (ICRA),Stockholm:IEEE,2016:3095-3102.
[5] HBALI Y,HBALI S,BALLIHI L,et al. Skeleton-based human activity recognition for elderly monitoring systems[J]. IET Computer Vision,2017,12(1):16-26.
[6] PRESTI L L,LA C M. 3D skeleton-based human action classification:A survey[J]. Pattern Recognition,2016,53:130-147.
[7] EVANGELIDIS G,SINGH G,HORAUD R. Skeletal quads:Human action recognition using joint quadruples[C]//201422nd International Conference on Pattern Recognition,Stockholm:IEEE,2014:4513-4518.
[8] OFLI F,CHAUDHRY R,KURILLO G,et al. Sequence of the most informative joints (smij):A new representation for human skeletal action recognition[J]. Journal of Visual Communication and Image Representation,2014,25(1):24-38.
[9] WANG C,WANG Y,YUILLE A L. Mining 3d key-pose-motifs for action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas:IEEE,2016:2639-2647.
[10] KAPSOURAS I,NIKOLAIDIS N. Action recognition on motion capture data using a dynemes and forward differences representation[J]. Journal of Visual Communication and Image Representation,2014,25(6):1432-1445.
[11] HUSSEIN M E,TORKI M,GOWAYYED M A,et al. Human action recognition using a temporal hierarchy of covariance descriptors on 3D joint locations[C]//Twenty-Third International Joint Conference on Artificial Intelligence,Beijing:AAAI,2013:2466-2472.
[12] YANG X,TIAN Y L. Eigenjoints-based action recognition using naive-bayes-nearest-neighbor[C]//2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops,Rhode Island:IEEE,2012:14-19.
[13] DU Y,WANG W,WANG L. Hierarchical recurrent neural network for skeleton based action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Boston:IEEE,2015:1110-1118.
[14] LIU J,SHAHROUDY A,XU D,et al. Spatio-temporal lstm with trust gates for 3d human action recognition[C]//European Conference on Computer Vision,Amsterdam:Springer,2016:816-833.
[15] JAAKKOLA T,HAUSSLER D. Exploiting generative models in discriminative classifiers[C]//Advances in Neural Information Processing Systems,Denver:IEEE,1999:487-493.
[16] WANG C,FLYNN J,WANG Y,et al. Recognizing actions in 3d using action-snippets and activated simplices[C]//Thirtieth AAAI Conference on Artificial Intelligence,Phoenix:AAAI,2016:3604-3610.
[17] LIU B,JU Z,LIU H. A structured multi-feature representation for recognizing human action and interaction[J]. Neurocomputing,2018,318:287-296.
[18] ZANFIR M,LEORDEANU M,SMINCHISESCU C. The moving pose:An efficient 3d kinematics descriptor for low-latency action recognition and detection[C]//Proceedings of the IEEE International Conference on Computer Vision,Sydney:IEEE,2013:2752-2759.
[19] OHN-BAR E,TRIVEDI M. Joint angles similarities and HOG2 for action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops,Portland:IEEE,2013:465-470.
[20] Müller M,Röder T,Clausen M,et al. Documentation mocap database hdm05[R]. 2007. Technical Report CG-2007-2,Universität Bonn,2007.
[21] OFLI F,CHAUDHRY R,KURILLO G,et al. Berkeley mhad:A comprehensive multimodal human action database[C]//2013 IEEE Workshop on Applications of Computer Vision (WACV),Clearwater Beach:IEEE,2013:53-60.
[22] XIA L,CHEN C C,AGGARWAL J K. View invariant human action recognition using histograms of 3d joints[C]//2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops,Rhode Island:IEEE,2012:20-27.
[23] CHANG C C,LIN C J. LIBSVM:A library for support vector machines[J]. ACM Transactions on Intelligent Systems and Technology (TIST),2011,2(3):27.
[24] DENG L,LEUNG H,GU N,et al. Generalized model-based human motion recognition with body partition index maps[C]//Computer Graphics Forum,Oxford,UK:Blackwell Publishing Ltd,2012,31(1):202-215.
[25] OFLI F,CHAUDHRY R,KURILLO G,et al. Sequence of the most informative joints (SMIJ):A new representation for human skeletal action recognition[C]//2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops),Rhode Island:IEEE,2012:8-13.
[26] GOWAYYED M A,TORKI M,HUSSEIN M E,et al. Histogram of oriented displacements (HOD):Describing trajectories of human joints for action recognition[C]//Twenty-Third International Joint Conference on Artificial Intelligence,Beijing:AAAI,2013:1351-1357.
[27] VANTIGODI S,RADHAKRISHNAN V B. Action recognition from motion capture data using meta-cognitive RBF network classifier[C]//2014 IEEE Ninth International Conference on Intelligent Sensors,Sensor Networks and Information Processing (ISSNIP),Singapore:IEEE,2014:1-6.

面向动作识别的旋转投影变换关节特征骨架描述

Rotation and Projection Transformed Joint Feature Descriptors of Human Skeletal Action Recognition

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	唐敏杰, 赵欢, 丁汉. 二进制点云局部特征描述子研究[J]. 机械工程学报, 2021, 57(2): 219-229.
[2]	王志学, 刘献礼, 李茂月, LIANG S Y, 王力翚, 李玉强, 孟博洋. 切削加工颤振智能监控技术[J]. 机械工程学报, 2020, 56(24): 1-23.
[3]	陈鲲, 茆志伟, 张进杰, 江志农. 基于和声搜索优化栈式自编码器的柴油发动机故障诊断[J]. 机械工程学报, 2020, 56(11): 132-140.
[4]	崔玲丽, 王鑫, 王华庆, 胥永刚, 张建宇. 基于改进开关卡尔曼滤波的轴承故障特征提取方法[J]. 机械工程学报, 2019, 55(7): 44-51.
[5]	王华庆, 任帮月, 宋浏阳, 董方, 王梦阳. 基于终止准则改进K-SVD字典学习的稀疏表示特征增强方法[J]. 机械工程学报, 2019, 55(7): 35-43.
[6]	郭莹莹, 赵学智, 上官文斌, 李伟光. 柔性椭圆轴承故障频率分析与CMWT-FH特征提取[J]. 机械工程学报, 2019, 55(23): 154-161.
[7]	顾晓辉, 杨绍普, 刘永强, 任彬, 张建超. 基于多目标交叉熵优化的轮对轴承故障特征提取方法[J]. 机械工程学报, 2018, 54(4): 285-292.
[8]	董玉德, 张荣团, 宋忠辉, 房玉强, 白苏诚, 张方亮. 胎面花纹边界特征提取及重构方法研究[J]. 机械工程学报, 2018, 54(3): 137-147.
[9]	孙健, 李洪儒. 一种基于复合谱与关联熵融合的特征提取方法[J]. 机械工程学报, 2017, 53(24): 96-103.
[10]	黎敏, 杨孟瑶, 陈泽, 赵启东. 基于局部特征提取的有监督的流形学习方法[J]. 机械工程学报, 2017, 53(24): 211-220.
[11]	马萍, 张宏立, 范文慧. 基于局部与全局结构保持算法的滚动轴承故障诊断^*[J]. 机械工程学报, 2017, 53(2): 20-25.
[12]	孟宗, 王亚超, 胡猛. 基于改进LMD和IED-SampEn的齿轮故障特征提取方法[J]. 机械工程学报, 2016, 52(5): 169-174.
[13]	徐科, 王磊, 王璟瑜. 基于Tetrolet变换的热轧钢板表面缺陷识别方法[J]. 机械工程学报, 2016, 52(4): 13-19.
[14]	周鹏, 徐科, 刘顺华. 基于剪切波和小波特征融合的金属表面缺陷识别方法[J]. 机械工程学报, 2015, 51(6): 98-103.
[15]	王建国, 李健, 万旭东. 基于奇异值分解和局域均值分解的滚动轴承故障特征提取方法[J]. 机械工程学报, 2015, 51(3): 104-110.