Free Gait Planning for a Hexapod Robot Based on Reinforcement Learning

doi:10.3901/JME.2019.05.036

Abstract

Abstract: In order to solve the problem of gait planning for hexapod robots and achieve the optimization and learning of free gaits on specific terrains, a discrete gait model is built based on the discretization of strides and the fusion of CPG mode and reflect model. Through the analysis of robot stability and the study of gait planning strategies, the complex gait planning problem is transformed into the reorder problem of states with the interval of oscillation period. Inspired by this idea, a free gait planning method is proposed from a new perspective. Then a gait model based on reinforcement learning is constructed based on the discretization of gait sequences to imitate the learning behaviour of biological gaits. And using the average stability margin as performance index, a free gait planning method based on reinforcement learning is proposed by developing the adjustment strategies of dynamic conversion probabilities between discrete gait units. The prototype gait experiment results show that both the free gait planning method and the free gait planning method based on reinforcement learning can generate free gaits in line with the laws of biological movements, and the free gait planning method based on reinforcement learning can achieve the optimization and learning of free gaits on specific terrains using gait history information.

Key words: discretization, free gait, gait planning, hexapod robot, reinforcement learning

CLC Number:

TP242

LI Manhong, ZHANG Minglu, ZHANG Jianhua, TIAN Ying, MA Yanyue. Free Gait Planning for a Hexapod Robot Based on Reinforcement Learning[J]. Journal of Mechanical Engineering, 2019, 55(5): 36-44.

References

[1] DENG Hua,XIN Guiyang,ZHONG Guoliang,et al. Gait and trajectory rolling planning and control of hexapod robots for disaster rescue applications[J]. Robotics and Autonomous Systems,2017,95(4):13-24.
[2] CHEN Xianbao,GAO Feng. Energy expenditure of trotting gait under different gait parameters[J]. Chinese Journal of Mechanical Engineering,2017,30(4):1-8.
[3] XU Yilin,GAO Feng,PAN Yang,et al. Method for six-legged robot stepping on obstacles by indirect force estimation[J]. Chinese Journal of Mechanical Engineering,2016,29(4):669-679.
[4] 丁希仑,王志英,ALBERTO R. 六边形对称分布六腿机器人的典型步态及其运动性能分析[J]. 机器人,2010,32(6):759-765. DING Xilun,WANG Zhiying,ALBERTO R. Typical gaits and motion analysis of a hexagonal symmetrical hexapod robot[J]. Robot,2010,32(6):759-765.
[5] 甄伟鲲,康熙,戴建生,等. 一种新型四足变胞爬行机器人的步态规划研究[J]. 机械工程学报,2016,52(11):26-33. ZHEN Weikun,KANG Xi,DAI Jiansheng,et al. Gait planning of a novel metamorphic quadruped robot[J]. Journal of Mechanical Engineering,2016,52(11):26-33.
[6] 李满宏,张明路,张建华,等. 基于离散化的六足机器人自由步态生成算法[J]. 机械工程学报,2016,52(3):18-25. LI Manhong,ZHANG Minglu,ZHANG Jianhua,et al. Free gait generation algorithm for a hexapod robot based on discretization[J]. Journal of Mechanical Engineering,2016,52(3):18-25.
[7] ERDEN M S,KEMAL L. Free gait generation with reinforcement learning for a six-legged robot[J]. Robotics and Autonomous Systems,2008,56(3):199-212.
[8] 王刚,张立勋,王立权. 仿蟹机器人交错等相位波形步态研究[J]. 机器人,2011,33(1):237-243. WANG Gang,ZHANG Lixun,WANG Liquan. On alternating equal-phase wave gait of crab-like robot[J]. Robot,2011,33(1):237-243.
[9] CARVALHO C S,RAPTOPOULOS L S,ANDRADE W S,et al. Extending SMER-based CPGs to accommodate total support phases and kinematics-safe transitions between gait rhythms of hexapod robots[J]. Neurocomputing,2015,170:113-127.
[10] ZHU Yaguang,GUO Tong,LIU Qiong,et al. A study of arbitrary gait pattern generation for turning of a bio-inspired hexapod robot[J]. Robotics and Autonomous Systems,2017,97(2):125-135.
[11] GRZELCZYK D,STAN C B,AWREJCEWICZ J. Prototype,control system architecture and controlling of the hexapod legs with nonlinear stick-slip[J]. Mechatronics,2016,37:63-78.
[12] LEI Jingtao,YU Huangying,WANG Tianmiao. Dynamic bending of bionic flexible body driven by pneumatic artificial muscles (PAMs) for spinning gait of quadruped robot[J]. Chinese Journal of Mechanical Engineering,2016,29(1):11-20.
[13] PORTA J M,CELAYA E. Reactive free-gait generation to follow arbitrary trajectories with a hexapod robot[J]. Robotics and Autonomous Systems,2004,47(4):187-201.
[14] ZHANG He,LIU Yubin,ZHAO Jie,et al. Development of a bionic hexapod robot for walking on unstructured terrain[J]. Journal of Bionic Engineering,2014,11(2):176-187.
[15] SANTOS C P,MATOS V. CPG modulation for navigation and omnidirectional quadruped locomotion[J]. Robotics and Autonomous Systems,2012,60(6):912-927.
[16] 黄博,姚玉峰,孙立宁. 基于中枢神经模式的四足机器人步态控制[J]. 机械工程学报,2010,46(7):1-6. HUANG Bo,YAO Yufeng,SUN Lining. Quadruped robot gait control based on central pattern generator[J]. Journal of Mechanical Engineering,2010,46(7):1-6.
[17] KALUZA P,CIOAC T. Phase oscillator neural network as artificial central pattern generator for robots[J]. Neurocomputing,2012,97:115-124.
[18] 孟健,李贻斌,柴汇,等. 连续不规则台阶环境四足机器人步态规划与控制[J]. 机器人,2015,37(1):85-93. MENG Jian,Li Yibin,CHAI Hui,et al. Gait planning and control of quadruped robots in continuous irregular steps environment[J]. Robot,2015,37(1):85-93.
[19] 张帅帅,荣学文,李贻斌,等. 崎岖地形环境下四足机器人的静步态规划方法[J]. 吉林大学学报,2016,46(4):1287-1296. ZHANG Shuaishuai,RONG Xuewen,LI Yibin,et al. Static gait planning method for quadruped robots on rough terrains[J]. Journal of Jilin University,2016,46(4):1287-1296.
[20] 李满宏,张建华,张小俊,等. 基于马尔可夫决策过程的六足机器人自由步态规划[J]. 机器人,2015,37(5):529-537. LI Manhong,ZHANG Jianhua,ZHANG Xiaojun,et al. Free gait planning for a hexapod robot based on Markov decision process[J]. Robot,2015,37(5):529-537.
[21] 韩宝玲,赵锐,罗庆生,等. 基于粒子群算法的四足机器人静步态优化方法[J]. 北京理工大学学报,2017,37(5):461-465. HAN Baoling,ZHAO Rui,LUO Qingsheng,et al. Static gait optimization method for quadruped robot based on particle swarm optimization algorithm[J]. Transactions of Beijing Institute of Technology,2017,37(5):461-465.
[22] AGHELI M,NESTINGER S S. Force-based stability margin for multi-legged robots[J]. Robotics and Autonomous Systems,2016,83(2):138-149.