基于增强学习的六足机器人自由步态规划

doi:10.3901/JME.2019.05.036

机械工程学报 ›› 2019, Vol. 55 ›› Issue (5): 36-44.doi: 10.3901/JME.2019.05.036

基于增强学习的六足机器人自由步态规划

李满宏, 张明路, 张建华, 田颖, 马艳悦

河北工业大学机械工程学院天津 300130

收稿日期:2018-05-11 修回日期:2018-12-28 出版日期:2019-03-05 发布日期:2019-03-05
通讯作者: 田颖(通信作者),女,1987年出生,博士,讲师。主要研究方向为机器人动力学与运动控制。E-mail:flyserelo@126.com.cn
作者简介:李满宏,男,1987年出生,博士,讲师。主要研究方向为六足机器人步态规划与运动控制。E-mail:lmh9181219@163.com.cn;张明路,男,1964年出生,博士,教授,博士研究生导师。主要研究方向为智能机器人技术。E-mail:zhangml@hebut.edu.cn;张建华,男,1979年出生,博士,教授,博士研究生导师。主要研究方向为机器人柔性控制与安全作业方法。E-mail:jhzhang@hebut.edu.cn;马艳悦,男,1993年出生,硕士研究生。主要研究方向为机器人关节柔性控制方法。E-mail:Mayuanyue@hebut.edu.cn
基金资助:
国家自然科学基金（61803142，61473113，61503119）、河北省自然科学基金（F2018202210）和河北省高等学校科学技术研究（QN2017047）资助项目。

Free Gait Planning for a Hexapod Robot Based on Reinforcement Learning

LI Manhong, ZHANG Minglu, ZHANG Jianhua, TIAN Ying, MA Yanyue

School of Mechanical Engineering, Hebei University of Technology, Tianjin 300130

Received:2018-05-11 Revised:2018-12-28 Online:2019-03-05 Published:2019-03-05

摘要/Abstract

摘要： 为解决六足机器人步态规划问题，实现特定地形上机器人自由步态的优化学习，基于机器人单足步距的离散化处理，融合CPG模型的时间节拍原理与反射模型的空间规则约束机制，构建六足机器人离散化步态模型。通过机器人稳定性分析与步态规划策略研究，将复杂的步态规划问题转化为以振荡周期为时间间隔的位置状态间的排序问题，从新的视角提出了一种六足机器人自由步态规划的基本框架与方法。在此基础上，模仿生物步态的学习行为，基于步态序列的离散化处理，构建了基于增强学习的步态模型，并以机器人平均稳定裕量为优化目标，通过制定步态离散单元间动态转换概率的调整策略，提出了基于增强学习的自由步态规划方法。样机试验显示，自由步态规划方法与基于增强学习的自由步态规划方法均可规划出相对符合生物步态行为特征的稳定自由步态，且后者可利用步态历史信息实现特定地形上自由步态的优化学习。

关键词: 步态规划, 离散化, 六足机器人, 增强学习, 自由步态

Abstract: In order to solve the problem of gait planning for hexapod robots and achieve the optimization and learning of free gaits on specific terrains, a discrete gait model is built based on the discretization of strides and the fusion of CPG mode and reflect model. Through the analysis of robot stability and the study of gait planning strategies, the complex gait planning problem is transformed into the reorder problem of states with the interval of oscillation period. Inspired by this idea, a free gait planning method is proposed from a new perspective. Then a gait model based on reinforcement learning is constructed based on the discretization of gait sequences to imitate the learning behaviour of biological gaits. And using the average stability margin as performance index, a free gait planning method based on reinforcement learning is proposed by developing the adjustment strategies of dynamic conversion probabilities between discrete gait units. The prototype gait experiment results show that both the free gait planning method and the free gait planning method based on reinforcement learning can generate free gaits in line with the laws of biological movements, and the free gait planning method based on reinforcement learning can achieve the optimization and learning of free gaits on specific terrains using gait history information.

Key words: discretization, free gait, gait planning, hexapod robot, reinforcement learning

中图分类号:

TP242

李满宏, 张明路, 张建华, 田颖, 马艳悦. 基于增强学习的六足机器人自由步态规划[J]. 机械工程学报, 2019, 55(5): 36-44.

LI Manhong, ZHANG Minglu, ZHANG Jianhua, TIAN Ying, MA Yanyue. Free Gait Planning for a Hexapod Robot Based on Reinforcement Learning[J]. Journal of Mechanical Engineering, 2019, 55(5): 36-44.

参考文献

[1] DENG Hua,XIN Guiyang,ZHONG Guoliang,et al. Gait and trajectory rolling planning and control of hexapod robots for disaster rescue applications[J]. Robotics and Autonomous Systems,2017,95(4):13-24.
[2] CHEN Xianbao,GAO Feng. Energy expenditure of trotting gait under different gait parameters[J]. Chinese Journal of Mechanical Engineering,2017,30(4):1-8.
[3] XU Yilin,GAO Feng,PAN Yang,et al. Method for six-legged robot stepping on obstacles by indirect force estimation[J]. Chinese Journal of Mechanical Engineering,2016,29(4):669-679.
[4] 丁希仑,王志英,ALBERTO R. 六边形对称分布六腿机器人的典型步态及其运动性能分析[J]. 机器人,2010,32(6):759-765. DING Xilun,WANG Zhiying,ALBERTO R. Typical gaits and motion analysis of a hexagonal symmetrical hexapod robot[J]. Robot,2010,32(6):759-765.
[5] 甄伟鲲,康熙,戴建生,等. 一种新型四足变胞爬行机器人的步态规划研究[J]. 机械工程学报,2016,52(11):26-33. ZHEN Weikun,KANG Xi,DAI Jiansheng,et al. Gait planning of a novel metamorphic quadruped robot[J]. Journal of Mechanical Engineering,2016,52(11):26-33.
[6] 李满宏,张明路,张建华,等. 基于离散化的六足机器人自由步态生成算法[J]. 机械工程学报,2016,52(3):18-25. LI Manhong,ZHANG Minglu,ZHANG Jianhua,et al. Free gait generation algorithm for a hexapod robot based on discretization[J]. Journal of Mechanical Engineering,2016,52(3):18-25.
[7] ERDEN M S,KEMAL L. Free gait generation with reinforcement learning for a six-legged robot[J]. Robotics and Autonomous Systems,2008,56(3):199-212.
[8] 王刚,张立勋,王立权. 仿蟹机器人交错等相位波形步态研究[J]. 机器人,2011,33(1):237-243. WANG Gang,ZHANG Lixun,WANG Liquan. On alternating equal-phase wave gait of crab-like robot[J]. Robot,2011,33(1):237-243.
[9] CARVALHO C S,RAPTOPOULOS L S,ANDRADE W S,et al. Extending SMER-based CPGs to accommodate total support phases and kinematics-safe transitions between gait rhythms of hexapod robots[J]. Neurocomputing,2015,170:113-127.
[10] ZHU Yaguang,GUO Tong,LIU Qiong,et al. A study of arbitrary gait pattern generation for turning of a bio-inspired hexapod robot[J]. Robotics and Autonomous Systems,2017,97(2):125-135.
[11] GRZELCZYK D,STAN C B,AWREJCEWICZ J. Prototype,control system architecture and controlling of the hexapod legs with nonlinear stick-slip[J]. Mechatronics,2016,37:63-78.
[12] LEI Jingtao,YU Huangying,WANG Tianmiao. Dynamic bending of bionic flexible body driven by pneumatic artificial muscles (PAMs) for spinning gait of quadruped robot[J]. Chinese Journal of Mechanical Engineering,2016,29(1):11-20.
[13] PORTA J M,CELAYA E. Reactive free-gait generation to follow arbitrary trajectories with a hexapod robot[J]. Robotics and Autonomous Systems,2004,47(4):187-201.
[14] ZHANG He,LIU Yubin,ZHAO Jie,et al. Development of a bionic hexapod robot for walking on unstructured terrain[J]. Journal of Bionic Engineering,2014,11(2):176-187.
[15] SANTOS C P,MATOS V. CPG modulation for navigation and omnidirectional quadruped locomotion[J]. Robotics and Autonomous Systems,2012,60(6):912-927.
[16] 黄博,姚玉峰,孙立宁. 基于中枢神经模式的四足机器人步态控制[J]. 机械工程学报,2010,46(7):1-6. HUANG Bo,YAO Yufeng,SUN Lining. Quadruped robot gait control based on central pattern generator[J]. Journal of Mechanical Engineering,2010,46(7):1-6.
[17] KALUZA P,CIOAC T. Phase oscillator neural network as artificial central pattern generator for robots[J]. Neurocomputing,2012,97:115-124.
[18] 孟健,李贻斌,柴汇,等. 连续不规则台阶环境四足机器人步态规划与控制[J]. 机器人,2015,37(1):85-93. MENG Jian,Li Yibin,CHAI Hui,et al. Gait planning and control of quadruped robots in continuous irregular steps environment[J]. Robot,2015,37(1):85-93.
[19] 张帅帅,荣学文,李贻斌,等. 崎岖地形环境下四足机器人的静步态规划方法[J]. 吉林大学学报,2016,46(4):1287-1296. ZHANG Shuaishuai,RONG Xuewen,LI Yibin,et al. Static gait planning method for quadruped robots on rough terrains[J]. Journal of Jilin University,2016,46(4):1287-1296.
[20] 李满宏,张建华,张小俊,等. 基于马尔可夫决策过程的六足机器人自由步态规划[J]. 机器人,2015,37(5):529-537. LI Manhong,ZHANG Jianhua,ZHANG Xiaojun,et al. Free gait planning for a hexapod robot based on Markov decision process[J]. Robot,2015,37(5):529-537.
[21] 韩宝玲,赵锐,罗庆生,等. 基于粒子群算法的四足机器人静步态优化方法[J]. 北京理工大学学报,2017,37(5):461-465. HAN Baoling,ZHAO Rui,LUO Qingsheng,et al. Static gait optimization method for quadruped robot based on particle swarm optimization algorithm[J]. Transactions of Beijing Institute of Technology,2017,37(5):461-465.
[22] AGHELI M,NESTINGER S S. Force-based stability margin for multi-legged robots[J]. Robotics and Autonomous Systems,2016,83(2):138-149.

基于增强学习的六足机器人自由步态规划

Free Gait Planning for a Hexapod Robot Based on Reinforcement Learning

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	张程煜, 郭盛, 赵福群. 新型轮腿复合机器人的运动分析及步态研究[J]. 机械工程学报, 2019, 55(15): 145-153.
[2]	杨雪锋, 郭振武, 王斌锐, 王凌, 金英连. 基于带反馈Hopf振荡器的六足机器人斜坡步态发生器设计[J]. 机械工程学报, 2018, 54(21): 41-48.
[3]	王雷, 邓宗全, 武练梅, 汪豪蒂. 压电圆环结构动刚度控制有限差分仿真分析[J]. 机械工程学报, 2018, 54(18): 115-122.
[4]	陈征, 刘亚辉, 杨芳. 基于进化-增强学习方法的插电式混合动力公交车能量管理策略[J]. 机械工程学报, 2017, 53(16): 86-93.
[5]	李满宏, 张明路, 张建华, 张小俊. 基于离散化的六足机器人自由步态生成算法[J]. 机械工程学报, 2016, 52(3): 18-25.
[6]	秦国华, 王子琨, 吴竹溪, 鲁宇明. 基于表面网格离散化与遗传算法的复杂工件装夹布局规划方法^*[J]. 机械工程学报, 2016, 52(13): 195-203.
[7]	甄伟鲲, 康熙, 张新生, 戴建生. 一种新型四足变胞爬行机器人的步态规划研究[J]. 机械工程学报, 2016, 52(11): 26-33.
[8]	刘逸群, 邓宗全, 刘振, 丁亮, 高海波, 李宇超. 液压驱动六足机器人一种低冲击运动规划方法[J]. 机械工程学报, 2015, 51(3): 10-17.
[9]	庄红超;高海波;邓宗全;丁亮;刘振. 电驱动重载六足机器人关节转速分析方法[J]. , 2013, 49(23): 44-52.
[10]	王立鹏;王军政;汪首坤;何玉东. 基于足端轨迹规划算法的液压四足机器人步态控制策略[J]. , 2013, 49(1): 39-44.
[11]	常勇;杨富富;胡志超;王沁峰;李毅;李延平. 作平面运动滚子从动件盘形凸轮机构的广义第II类机构综合问题[J]. , 2012, 48(15): 47-57.
[12]	于秀丽;魏世民;廖启征. 仿人机器人发展及其技术探索[J]. , 2009, 45(3): 71-75.
[13]	周金宇;谢里阳. 多状态系统共因失效机理与定量分析[J]. , 2008, 44(10): 77-82.
[14]	王安麟;石斌;赵群飞;吴仁智;陈凝. 多关节双足步行机器椅行走步态规划[J]. , 2007, 43(11): 168-172.
[15]	赵荣珍;张优云. 基于Rough set知识获取的故障数据表聚类离散化方法研究[J]. , 2005, 41(1): 145-150.