Motion planning for hexapod robot using deep reinforcement learning

Huiqiao FU; Kaiqiang TANG; Guizhou DENG; Xinpeng WANG; Chunlin CHEN

doi:10.11959/j.issn.2096-6652.202039

您当前的位置：

首页 >

文章列表页 >

Motion planning for hexapod robot using deep reinforcement learning

Special Issue: Deep Reinforcement Learning | 更新时间：2024-06-05

- Motion planning for hexapod robot using deep reinforcement learning
- Chinese Journal of Intelligent Science and Technology Vol. 2, Issue 4, Pages: 361-371(2020)
- 作者机构：
  
  1. 南京大学工程管理学院，江苏南京 210046
  2. 西南科技大学制造科学与工程学院，四川绵阳 621010
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-6652.202039
  CLC： TP242.6
- Published Online：2020-12，
  
  Published：15 December 2020
- 稿件说明：
移动端阅览
Huiqiao FU, Kaiqiang TANG, Guizhou DENG, et al. Motion planning for hexapod robot using deep reinforcement learning[J]. Chinese journal of intelligent science and technology, 2020, 2(4): 361-371.
DOI：

Huiqiao FU, Kaiqiang TANG, Guizhou DENG, et al. Motion planning for hexapod robot using deep reinforcement learning[J]. Chinese journal of intelligent science and technology, 2020, 2(4): 361-371. DOI： 10.11959/j.issn.2096-6652.202039.

摘要

六足机器人拥有多个冗余自由度，适用于复杂的非结构环境。离散环境作为非结构环境的一个苛刻特例，需要六足机器人具备更加高效可靠的运动策略。以平面随机梅花桩为例，设定随机起始点与目标区域，利用深度强化学习算法进行训练，并得到六足机器人在平面梅花桩环境中的运动策略。为了加快训练进程，采用具有优先经验重放机制的深度确定性策略梯度算法。最后在真实环境中进行验证，实验结果表明，所规划的运动策略能让六足机器人在平面梅花桩环境中高效平稳地从起始点运动到目标区域。为六足机器人在真实离散环境中的精确运动规划奠定了基础。

Abstract

Hexapod robot have multiple redundant degrees of freedom and are suitable for complex unstructured environments.Discrete environments

as a harsh special case of unstructured environments

require hexapod robots to have more efficient and reliable motion strategies.A plane random plum-blossom pile environment was taken as an example.A random starting point and a target area were set

and the deep reinforcement learning algorithm was applied to plan a motion strategy for a hexapod robot in theplane plum-blossompile environment.To speed up the training process

a deep deterministic policy gradient algorithm with a prioritized experience replay mechanism was used.Finally the policy was verified in a real environment.The results show that the planned motion strategy can make the hexapod robot move efficiently and smoothly from a starting point to a target area in aplane plum-blossom pile environment.This work lays the foundation for the precise motion planning of hexapod robots in the real discrete environment.

关键词

Keywords

references

李满宏 , 张明路 , 张建华 , 等 . 六足机器人关键技术综述 [J ] . 机械设计 , 2015 , 32 ( 10 ): 1 - 8 .

LI M H , ZHANG M L , ZHANG J H , et al . Review on key technology of the hexapod robot [J ] . Journal of Machine Design , 2015 , 32 ( 10 ): 1 - 8 .

STELZER A , HIRSCHMÜLLER H , GÖRNER M . Stereo-visionbased navigation of a six-legged walking robot in unknown rough terrain [J ] . The International Journal of Robotics Research , 2012 , 31 ( 4 ): 381 - 402 .

HOMBERGER T , BJELONIC M , KOTTEGE N , et al . Terrain-dependant control of hexapod robots using vision [M ] // International Symposium on Experimental Robotics . Cham : Springer , 2016 : 92 - 102 .

胡勇 . 六足机器人梅花桩行走步态研究 [D ] . 绵阳:西南科技大学 , 2019 .

HU Y . Study on gait of hexapod robot walking on plum pile [D ] . Mianyang:Southwest University of Science and Technology , 2019 .

ČÍŽEK P , MASRI D , FAIGL J . Foothold placement planning with a hexapod crawling robot [C ] // 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2017 : 4096 - 4101 .

MOSTAFA K , CHIANG J Y , WEI K T , et al . Image-based method for determining better walking strategies for hexapods [J ] . International Journal of Advanced Robotic Systems , 2015 , 12 ( 5 ): 58 .

沈宇 , 韩金朋 , 李灵犀 , 等 . 游戏智能中的 AI——从多角色博弈到平行博弈 [J ] . 智能科学与技术学报 , 2020 , 2 ( 3 ): 205 - 213 .

SHEN Y , HAN J P , LI L X , et al . AI in game intelligence—from multi-role game to parallel game [J ] . Chinese Journal of Intelligent Science and Technology , 2020 , 2 ( 3 ): 205 - 213 .

王飞跃 , 曹东璞 , 魏庆来 . 强化学习:迈向知行合一的智能机制与算法 [J ] . 智能科学与技术学报 , 2020 , 2 ( 2 ): 101 - 106 .

WANG F Y , CAO D P , WEI Q L . Reinforcement learning:toward action-knowledge merged intelligent mechanisms and algorithms [J ] . Chinese Journal of Intelligent Science and Technology , 2020 , 2 ( 2 ): 101 - 106 .

刘全 , 翟建伟 , 章宗长 , 等 . 深度强化学习综述 [J ] . 计算机学报 , 2018 , 41 ( 1 ): 1 - 27 .

LIU Q , ZHAI J W , ZHANG Z Z , et al . A survey on deep reinforcement learning [J ] . Chinese Journal of Computers , 2018 , 41 ( 1 ): 1 - 27 .

卜祥津 . 基于深度强化学习的未知环境下机器人路径规划的研究 [D ] . 哈尔滨:哈尔滨工业大学 , 2018 .

BU X J . Research on robot path planning under unknown environment based on deep reinforcement learning [D ] . Harbin:Harbin Institute of Technology , 2018 .

孙长银 , 穆朝絮 . 多智能体深度强化学习的若干关键科学问题 [J ] . 自动化学报 , 2020 ( 7 ): 1301 - 1312 .

SUN C Y , MU C X . Important scientific problems of multi-agent deep reinforcement learning [J ] . Acta Automatica Sinica , 2020 ( 7 ): 1301 - 1312 .

GARIBALDI JONATHAN M , 陈虹宇 , 李小双 . 差异与学习:模糊系统与模糊推理 [J ] . 智能科学与技术学报 , 2019 , 1 ( 4 ): 319 - 326 .

GARIBALDI JONATHAN M , CHEN H Y , LI X S . Variation and learning:fuzzy system and fuzzy inference [J ] . Chinese Journal of Intelligent Science and Technology , 2019 , 1 ( 4 ): 319 - 326 .

陈德旺 , 蔡际杰 , 黄允浒 . 面向可解释性人工智能与大数据的模糊系统发展展望 [J ] . 智能科学与技术学报 , 2019 , 1 ( 4 ): 327 - 334 .

CHEN D W , CAI J J , HUANG Y H . Development prospect of fuzzy system oriented to interpretable artificial intelligence and big data [J ] . Chinese Journal of Intelligent Science and Technology , 2019 , 1 ( 4 ): 327 - 334 .

WANG Z , CHEN C L , LI H X , et al . Incremental reinforcement learning with prioritized sweeping for dynamic environments [J ] . IEEE/ASME Transactions on Mechatronics , 2019 , 24 ( 2 ): 621 - 632 .

TSOUNIS V , ALGE M , LEE J , et al . Deepgait:planning and control of quadrupedal gaits using deep reinforcement learning [J ] . IEEE Robotics and Automation Letters , 2020 , 5 ( 2 ): 3699 - 3706 .

SHAHRIARI M , KHAYYAT A A . Gait analysis of a six-legged walking robot using fuzzy reward reinforcement learning [C ] // IEEE 13th Iranian Conference on Fuzzy Systems . Piscataway:IEEE Press , 2013 : 1 - 4 .

唐开强 . 基于迁移强化学习的六足机器人步态学习研究 [D ] . 南京:南京大学 , 2019 .

TANG K Q . Research on gait learning of hexapod robot based on transfer reinforcement learning [D ] . Nanjing:Nanjing University , 2019 .

HOU Y N , LIU L F , WEI Q , et al . A novel DDPG method with prioritized experience replay [C ] // 2017 IEEE International Conference on Systems,Man,and Cybernetics . Piscataway:IEEE Press , 2017 : 316 - 321 .

REN Z P , DONG D Y , LI H X , et al . Self-paced prioritized curriculum learning with coverage penalty in deep reinforcement learning [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2018 , 29 ( 6 ): 2216 - 2226 .

SHI W J , SONG S J , WU H , et al . Regularizedanderson acceleration for off-policy deep reinforcement learning [J ] . arXiv preprint , 2019 ,arXiv:1909. 03245 .

TING L H , BLICKHAN R , FULL R J . Dynamic and static stability in hexapedal runners [J ] . Journal of Experimental Biology , 1994 , 197 ( 1 ): 251 - 269 .

SUTTON R , BARTO A . Reinforcement learning:an introduction [M ] . Cambridge : MIT Press , 1998 .

WATKINS C . Learning from delayed rewards [J ] . PhD thesis,University of Cambridge , 1989 .

SILVER D , LEVER G , HEESS N , et al . Deterministic policy gradient algorithms [C ] // The 31st International Conference on Machine Learning . New York:ACM Press , 2014 : 387 - 395 .

IOFFE S , SZEGEDY C . Batch normalization:accelerating deep network training by reducing internal covariate shift [C ] // Proceedings of Machine Learning Research .[S.l.:s.n. ] , 2015 , 37 448 - 456 .

LILLICRAP T P , HUNT J J , PRITZEL A , et al . Continuous control with deep reinforcement learning [J ] . arXiv preprint , 2015 ,arXiv:1509. 02971 .

SCHAUL T , QUAN J , ANTONOGLOU I , et al . Prioritized experience replay [J ] . arXiv preprint , 2015 ,arXiv:1511. 05952 .

MAHMOOD A R , HASSELT H P , SUTTON R S . Weighted importance sampling for off-policy learning with linear function approximation [C ] // Conference and Workshop on Neural Information Processing Systems . New York:ACM Press , 2014 : 3014 - 3022 .

Views

269

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Review: the application of deep reinforcement learning to quantitative trading in financial market

Research on OAC model for quantitative trading of digital currency

HVAC model-free optimal control method based on double-pools DQN

TD3-based energy management strategy for hybrid energy storage system of electric vehicle

Overview of intelligent game:enlightenment of game AI to combat deduction

Related Author

LI Xiangxia

WEN Jiancheng

HE Yijun

XU Bo

LI Xiangxia

HE Yijun

XU Bo

You LU

Related Institution

Guangdong Intelligent Business Engineering Technology Research Center

School of Information, Guangdong University of Finance & Economics

School of Information Science, Guangdong University of Finance & Economics

Key Laboratory of Performance Evolution and Control for Engineering Structures of Ministry of Education, Tongji University

School of Mechanical Engineering, Tongji University

Postal code：100079
Tel：（010）53879206 Email：cjist@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：35685 Visits Today：160

⁰