Rotation and Projection Transformed Joint Feature Descriptors of Human Skeletal Action Recognition

doi:10.3901/JME.2021.05.019

Abstract

Abstract: In robotics, human skeletal action recognition with joint-based representations are widely used in human-robot collaboration. Mining temporal and spatial information in action sequence is a challenging problem. Insufficient information extraction tends to bring the algorithm less discriminability. In view of this, a high-performance joint-based descriptor called rotational and projective skeleton signature (RPSS) is proposed. First of all, build person-centric coordinate system; then learn average limbs (skeleton segments) lengths from the training set; Secondly, rotate the skeleton by a set of angles, meanwhile project the rotated skeletons on a certain plane to get the sequence-features; Finally, use histogram of oriented gradient (HOG) to exact spatio-tempral information to generate a final RPSS description. The RPSS descriptors were tested on a number of public datasets and compared with classical and learning based description algorithms. The results show that it has decent discriminative power, and the experiments adding different degrees of noises and recording average calculation time show it also has strong robustness and real-time performance.

Key words: skeleton, feature extraction, rotation & projection, human action recognition

CLC Number:

TP273

WU Xiaokang, ZHAO Huan, TANG Minjie, DING Han. Rotation and Projection Transformed Joint Feature Descriptors of Human Skeletal Action Recognition[J]. Journal of Mechanical Engineering, 2021, 57(5): 19-30.

References

[1] AGGARWAL J K,RYOO M S. Human activity analysis:A review[J]. ACM Computing Surveys (CSUR),2011,43(3):16.
[2] VEMULAPALLI R,ARRATE F,CHELLAPPA R. Human action recognition by representing 3D skeletons as points in a lie group[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Ohio:IEEE Computer Society,2014:588-595.
[3] ROITBERG A,SOMANI N,PERZYLO A,et al. Multimodal human activity recognition for industrial manufacturing processes in robotic workcells[C]//Proceedings of the 2015 ACM on International Conference on Multimodal Interaction,Seattle:ACM,2015:259-266.
[4] LIU C,TOMIZUKA M. Algorithmic safety measures for intelligent industrial co-robots[C]//2016 IEEE International Conference on Robotics and Automation (ICRA),Stockholm:IEEE,2016:3095-3102.
[5] HBALI Y,HBALI S,BALLIHI L,et al. Skeleton-based human activity recognition for elderly monitoring systems[J]. IET Computer Vision,2017,12(1):16-26.
[6] PRESTI L L,LA C M. 3D skeleton-based human action classification:A survey[J]. Pattern Recognition,2016,53:130-147.
[7] EVANGELIDIS G,SINGH G,HORAUD R. Skeletal quads:Human action recognition using joint quadruples[C]//201422nd International Conference on Pattern Recognition,Stockholm:IEEE,2014:4513-4518.
[8] OFLI F,CHAUDHRY R,KURILLO G,et al. Sequence of the most informative joints (smij):A new representation for human skeletal action recognition[J]. Journal of Visual Communication and Image Representation,2014,25(1):24-38.
[9] WANG C,WANG Y,YUILLE A L. Mining 3d key-pose-motifs for action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas:IEEE,2016:2639-2647.
[10] KAPSOURAS I,NIKOLAIDIS N. Action recognition on motion capture data using a dynemes and forward differences representation[J]. Journal of Visual Communication and Image Representation,2014,25(6):1432-1445.
[11] HUSSEIN M E,TORKI M,GOWAYYED M A,et al. Human action recognition using a temporal hierarchy of covariance descriptors on 3D joint locations[C]//Twenty-Third International Joint Conference on Artificial Intelligence,Beijing:AAAI,2013:2466-2472.
[12] YANG X,TIAN Y L. Eigenjoints-based action recognition using naive-bayes-nearest-neighbor[C]//2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops,Rhode Island:IEEE,2012:14-19.
[13] DU Y,WANG W,WANG L. Hierarchical recurrent neural network for skeleton based action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Boston:IEEE,2015:1110-1118.
[14] LIU J,SHAHROUDY A,XU D,et al. Spatio-temporal lstm with trust gates for 3d human action recognition[C]//European Conference on Computer Vision,Amsterdam:Springer,2016:816-833.
[15] JAAKKOLA T,HAUSSLER D. Exploiting generative models in discriminative classifiers[C]//Advances in Neural Information Processing Systems,Denver:IEEE,1999:487-493.
[16] WANG C,FLYNN J,WANG Y,et al. Recognizing actions in 3d using action-snippets and activated simplices[C]//Thirtieth AAAI Conference on Artificial Intelligence,Phoenix:AAAI,2016:3604-3610.
[17] LIU B,JU Z,LIU H. A structured multi-feature representation for recognizing human action and interaction[J]. Neurocomputing,2018,318:287-296.
[18] ZANFIR M,LEORDEANU M,SMINCHISESCU C. The moving pose:An efficient 3d kinematics descriptor for low-latency action recognition and detection[C]//Proceedings of the IEEE International Conference on Computer Vision,Sydney:IEEE,2013:2752-2759.
[19] OHN-BAR E,TRIVEDI M. Joint angles similarities and HOG2 for action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops,Portland:IEEE,2013:465-470.
[20] Müller M,Röder T,Clausen M,et al. Documentation mocap database hdm05[R]. 2007. Technical Report CG-2007-2,Universität Bonn,2007.
[21] OFLI F,CHAUDHRY R,KURILLO G,et al. Berkeley mhad:A comprehensive multimodal human action database[C]//2013 IEEE Workshop on Applications of Computer Vision (WACV),Clearwater Beach:IEEE,2013:53-60.
[22] XIA L,CHEN C C,AGGARWAL J K. View invariant human action recognition using histograms of 3d joints[C]//2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops,Rhode Island:IEEE,2012:20-27.
[23] CHANG C C,LIN C J. LIBSVM:A library for support vector machines[J]. ACM Transactions on Intelligent Systems and Technology (TIST),2011,2(3):27.
[24] DENG L,LEUNG H,GU N,et al. Generalized model-based human motion recognition with body partition index maps[C]//Computer Graphics Forum,Oxford,UK:Blackwell Publishing Ltd,2012,31(1):202-215.
[25] OFLI F,CHAUDHRY R,KURILLO G,et al. Sequence of the most informative joints (SMIJ):A new representation for human skeletal action recognition[C]//2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops),Rhode Island:IEEE,2012:8-13.
[26] GOWAYYED M A,TORKI M,HUSSEIN M E,et al. Histogram of oriented displacements (HOD):Describing trajectories of human joints for action recognition[C]//Twenty-Third International Joint Conference on Artificial Intelligence,Beijing:AAAI,2013:1351-1357.
[27] VANTIGODI S,RADHAKRISHNAN V B. Action recognition from motion capture data using meta-cognitive RBF network classifier[C]//2014 IEEE Ninth International Conference on Intelligent Sensors,Sensor Networks and Information Processing (ISSNIP),Singapore:IEEE,2014:1-6.