基于改进TD3算法的机械臂智能规划方法研究

张强; 文闻; 周晓东; 刘维惠; 初晓昱

doi:10.11959/j.issn.2096-6652.202225

您当前的位置：

首页 >

文章列表页 >

基于改进TD3算法的机械臂智能规划方法研究

专题：自主智能体灵巧精准操作学习 | 更新时间：2024-06-05

- 基于改进TD3算法的机械臂智能规划方法研究
- Research on the manipulator intelligent trajectory planning method based on the improved TD3 algorithm
- 智能科学与技术学报 2022年4卷第2期页码：223-232
- 作者机构：
- 作者简介：
  
  [ "张强（1978− ），男，博士，北京控制工程研究所机电技术事业部部长，中国航天科技集团有限公司科技委惯性专业组成员、研究员，主要研究方向为航天器各类驱动机构、执行机构、空间操控机械臂等" ]
  [ "文闻（1985− ），男，博士，北京控制工程研究所高级工程师，主要研究方向为空间机械臂运动规划与智能操控技术" ]
  [ "周晓东（1983− ），男，博士，北京控制工程研究所高级工程师，主要研究方向为仿生机器人、机器人交互控制技术" ]
  [ "刘维惠（1988− ），女，博士，北京控制工程研究所高级工程师，主要研究方向为空间机械臂智能交互技术" ]
  [ "初晓昱（1992− ），女，博士，北京控制工程研究所工程师，主要研究方向为空间机械臂智能规划、地外行星探测机器人控制技术" ]
- 基金信息：
  
  科技创新2030—“新一代人工智能”重大项目;The National Key Research and Development Program of China(2018AAA0103004)
- DOI：10.11959/j.issn.2096-6652.202225
  中图分类号： TP183
- 网络出版日期：2022-06，
  
  纸质出版日期：2022-06-15
- 稿件说明：
移动端阅览
张强, 文闻, 周晓东, 等. 基于改进TD3算法的机械臂智能规划方法研究[J]. 智能科学与技术学报, 2022,4(2):223-232.

Qiang ZHANG, Wen WEN, Xiaodong ZHOU, et al. Research on the manipulator intelligent trajectory planning method based on the improved TD3 algorithm[J]. Chinese journal of intelligent science and technology, 2022, 4(2): 223-232.
张强, 文闻, 周晓东, 等. 基于改进TD3算法的机械臂智能规划方法研究[J]. 智能科学与技术学报, 2022,4(2):223-232. DOI： 10.11959/j.issn.2096-6652.202225.

Qiang ZHANG, Wen WEN, Xiaodong ZHOU, et al. Research on the manipulator intelligent trajectory planning method based on the improved TD3 algorithm[J]. Chinese journal of intelligent science and technology, 2022, 4(2): 223-232. DOI： 10.11959/j.issn.2096-6652.202225.

摘要

针对某卫星搭载的4自由度机械臂轨迹规划问题，提出了一种基于改进的双延迟深度确定性策略梯度（TD3）算法的智能规划方法。该方法采用分阶段训练策略，在预训练阶段，采用了目标位置引导联合TD3算法进行轨迹优化的混合规划策略，训练结束后规划算法能够在机械臂关节空间对任意起点、终点进行速度轨迹的自主规划。这种目标引导机制减少了训练时不必要的探索，在一定程度上解决了高维动作空间中学习效率低下的问题。在二次训练阶段，首先通过示教获得一条无碰撞的安全参考轨迹，然后在训练过程中不断对这条轨迹进行模仿，使得最终算法输出的轨迹具备避障能力。

Abstract

An intelligent trajectory planning and obstacle avoidance method based on the improved twin delayed deep deterministic policy gradient algorithm (TD3) was proposed to solve the trajectory planning problem for a 4-DOF manipulator mounted on a satellite.The training strategy had 2 periods.In the pre-training stage

the target position was always guided combining with the output of the strategy network to optimize the trajectory.After the pre-training

the algorithm can autonomously output the velocity trajectory while the initial position and the target were specified randomly in the joint space of the manipulator.This target-guided mechanism decreased the unnecessary explorations and improved the learning efficiency in high dimensional action space.In the second training stage

a collision-free safety reference trajectory was firstly obtained by demonstration

and then this trajectory was constantly learned during the training process until the final output trajectory has the ability to avoid obstacles.

关键词

Keywords

references

胡照 , 王敏 , 袁俊刚 . 国外全电推进卫星平台的发展及启示 [J ] . 航天器环境工程 , 2015 , 32 ( 5 ): 566 - 570 .

HU Z , WANG M , YUAN J G . A review of the development of all-electric propulsion platform in the world [J ] . Spacecraft Environment Engineering , 2015 , 32 ( 5 ): 566 - 570 .

DONALD B , LYNCH K , RUS D . Algorithmic and computational robotics [M ] .[S.l. ] : A K Peters/CRC Press , 2001 .

KAVRAKI L E , SVESTKA P , LATOMBE J C , et al . Probabilistic roadmaps for path planning in high-dimensional configuration spaces [J ] . IEEE Transactions on Robotics and Automation , 1996 , 12 ( 4 ): 566 - 580 .

AMARJYOTI S . Deep reinforcement learning for robotic manipulation-the state of the art [J ] . arXiv preprint,2017,arXiv:1701.08878 .

多南讯 , 吕强 , 林辉灿 , 等 . 迈进高维连续空间:深度强化学习在机器人领域中的应用 [J ] . 机器人 , 2019 , 41 ( 2 ): 276 - 288 .

DUO N X , LYU Q , LIN H C , et al . Step into high-dimensional and continuous action space:a survey on applications of deep reinforcement learning to robotics [J ] . Robot , 2019 , 41 ( 2 ): 276 - 288 .

GU S X , HOLLY E , LILLICRAP T , et al . Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates [C ] // Proceedings of 2017 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2017 : 3389 - 3396 .

ZHU Y K , WANG Z Y , MEREL J , et al . Reinforcement and imitation learning for diverse visuomotor skills [J ] . arXiv preprint,2018,arXiv:1802.09564 .

SANGIOVANNI B , INCREMONA G P , PIASTRA M , et al . Self-configuring robot path planning with obstacle avoidance via deep reinforcement learning [J ] . IEEE Control Systems Letters , 2021 , 5 ( 2 ): 397 - 402 .

SANGIOVANNI B , RENDINIELLO A , INCREMONA G P , et al . Deep reinforcement learning for collision avoidance of robotic manipulators [C ] // Proceedings of 2018 European Control Conference . Piscataway:IEEE Press , 2018 .

KURUTACH T , CLAVERA I , DUAN Y , et al . Model-ensemble trust-region policy optimization [J ] . arXiv preprint,2018,arXiv:1802.10592 .

SCHULMAN J , WOLSKI F , DHARIWAL P , et al . Proximal policy optimization algorithms [J ] . arXiv preprint,2017,arXiv:1707.06347 .

ZHANG Y Z , CLAVERA I , TSAI B , et al . Asynchronous methods for model-based reinforcement learning [J ] . arXiv preprint,2019,arXiv:1910.12453 .

SILVER D , LEVER G , HEESS N , et al . Deterministic policy gradient algorithms [C ] // Proceedings of the 31st International Conference on Machine Learning .[S.l.:s.n. ] , 2014 : 387 - 395 .

LILLICRAP T P , HUNT J J , PRITZEL A , et al . Continuous control with deep reinforcement learning [J ] . arXiv preprint,2015,arXiv:1509.02971 .

BARTH-MARON G , HOFFMAN M W , BUDDEN D , et al . Distributed distributional deterministic policy gradients [J ] . arXiv preprint,2018,arXiv:1804.08617 .

FUJIMOTO S , VAN HOOF H , MEGER D . Addressing function approximation error in actor-critic methods [J ] . arXiv preprint,2018,arXiv:1802.09477 .

TASSA Y , DORON Y , MULDAL A , et al . DeepMind control suite [J ] . arXiv preprint,2018,arXiv:1801.00690 .

QURESHI A H , SIMEONOV A , BENCY M J , et al . Motion planning networks [C ] // Proceedings of 2019 International Conference on Robotics and Automation . Piscataway:IEEE Press , 2019 : 2118 - 2124 .

ZHONG J , WANG T , CHENG L L . Collision-free path planning for welding manipulator via hybrid algorithm of deep reinforcement learning and inverse kinematics [J ] . Complex ＆ Intelligent Systems , 2021 : 1 - 14 .

MIN C H , SONG J B . End-to-end robot manipulation using demonstration-guided goal strategie [C ] // Proceedings of 2019 16th International Conference on Ubiquitous Robots . Piscataway:IEEE Press , 2019 : 159 - 164 .

XU J , HOU Z M , WANG W , et al . Feedback deep deterministic policy gradient with fuzzy reward for robotic multiple peg-in-hole assembly tasks [C ] // Proceedings of the IEEE Transactions on Industrial Informatics . Piscataway:IEEE Press , 2019 : 1658 - 1667 .

ZHU Z Y , HU H S . Robot learning from demonstration in robotic assembly:a survey [J ] . Robotics , 2018 , 7 ( 2 ): 17 .

刘维惠 , 陈殿生 , 张立志 . 利用示教学习的移动机械臂轨迹避障算法 [J ] . 哈尔滨工程大学学报 , 2018 , 39 ( 9 ): 1546 - 1553 .

LIU W H , CHEN D S , ZHANG L Z . Learning from demonstration based obstacle avoidance algorithm to plan the trajectory of a mobile manipulator [J ] . Journal of Harbin Engineering University , 2018 , 39 ( 9 ): 1546 - 1553 .

ABDO N , KRETZSCHMAR H , SPINELLO L , et al . Learning manipulation actions from a few demonstrations [C ] // Proceedings of 2013 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2013 : 1268 - 1275 .

HEESS N , TB D , SRIRAM S , et al . Emergence of locomotion behaviours in rich environments [J ] . arXiv preprint,2017,arXiv:1707.02286 .

浏览量

915

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

暂无数据