An overview on algorithms and applications of deep reinforcement learning

Zhaoyang LIU; Chaoxu MU; Changyin SUN

doi:10.11959/j.issn.2096-6652.202034

您当前的位置：

首页 >

文章列表页 >

An overview on algorithms and applications of deep reinforcement learning

Special Issue: Deep Reinforcement Learning | 更新时间：2024-06-05

- An overview on algorithms and applications of deep reinforcement learning
- Chinese Journal of Intelligent Science and Technology Vol. 2, Issue 4, Pages: 314-326(2020)
- 作者机构：
  
  1. 天津大学电气自动化与信息工程学院，天津 300072
  2. 东南大学自动化学院，江苏南京 210096
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-6652.202034
  CLC： TP181
- Published Online：2020-12，
  
  Published：15 December 2020
- 稿件说明：
移动端阅览
Zhaoyang LIU, Chaoxu MU, Changyin SUN. An overview on algorithms and applications of deep reinforcement learning[J]. Chinese journal of intelligent science and technology, 2020, 2(4): 314-326.
DOI：

Zhaoyang LIU, Chaoxu MU, Changyin SUN. An overview on algorithms and applications of deep reinforcement learning[J]. Chinese journal of intelligent science and technology, 2020, 2(4): 314-326. DOI： 10.11959/j.issn.2096-6652.202034.

摘要

深度强化学习主要被用来处理感知-决策问题，已经成为人工智能领域重要的研究分支。概述了基于值函数和策略梯度的两类深度强化学习算法，详细阐述了深度Q网络、深度策略梯度及相关改进算法的原理，并综述了深度强化学习在视频游戏、导航、多智能体协作以及推荐系统等领域的应用研究进展。最后，对深度强化学习的算法和应用进行展望，针对一些未来的研究方向和研究热点给出了建议。

Abstract

Deep reinforcement learning (DRL) is mainly applied to solve the perception-decision problem

and has become an important research branch in the field of artificial intelligence.Two kinds of DRL algorithms based on value function and policy gradient were summarized

including deep Q network

policy gradient as well as related developed algorithms.In addition

the applications of DRL in video games

navigation

multi-agent cooperation and recommendation field were intensively reviewed.Finally

a prospect for the future research of DRL was made

and some research suggestions were given.

关键词

Keywords

references

SUTTON R S , BARTO A G . Reinforcement learning:an introduction [M ] . Cambridge : MIT Press , 2018 .

LECUN Y , BENGIO Y , HINTON G . Deep learning [J ] . Nature , 2015 , 521 ( 7553 ): 436 - 444 .

赵冬斌 , 邵坤 , 朱圆恒 , 等 . 深度强化学习综述：兼论计算机围棋的发展 [J ] . 控制理论与应用 , 2016 , 33 ( 6 ): 701 - 717 .

ZHAO D B , SHAO K , ZHU Y H , et al . Review of deep reinforcement learning and discussions on the development of computer Go [J ] . Control Theory ＆ Applications , 2016 , 33 ( 6 ): 701 - 717 .

万里鹏 , 兰旭光 , 张翰博 , 等 . 深度强化学习理论及其应用综述 [J ] . 模式识别与人工智能 , 2019 , 32 ( 1 ): 67 - 81 .

WAN L P , LAN X G , ZHANG H B , et al . A review of deep reinforcement learning theory and application [J ] . Pattern Recognition and Artificial Intelligence , 2019 , 32 ( 1 ): 67 - 81 .

MNIH V , KAVUKCUOGLU K , SILVER D , et al . Human-level control through deep reinforcement learning [J ] . Nature , 2015 , 518 ( 7540 ): 529 - 533 .

SILVER D , HUANG A , MADDISON C J , et al . Mastering the game of Go with deep neural networks and tree search [J ] . Nature , 2016 , 529 ( 7587 ): 484 - 489 .

SILVER D , SCHRITTWIESER J , SIMONYAN K , et al . Mastering the game of go without human knowledge [J ] . Nature , 2017 , 550 ( 7676 ): 354 - 359 .

BERNER C , BROCKMAN G , CHAN B , et al . Dota2 with large scale deep reinforcement learning [J ] . arXiv preprint , 2019 ,arXiv:1912. 06680 .

VINYALS O , BABUSCHKIN I , CZARNECKI W M , et al . Grandmaster level in StarCraftII using multi-agent reinforcement learning [J ] . Nature , 2019 , 575 ( 7782 ): 350 - 354 .

刘全 , 翟建伟 , 章宗长 , 等 . 深度强化学习综述 [J ] . 计算机学报 , 2018 , 41 ( 1 ): 1 - 27 .

LIU Q , ZHAI J W , ZHANG Z Z , et al . A survey on deep reinforcement learning [J ] . Chinese Journal of Computers , 2018 , 41 ( 1 ): 1 - 27 .

刘建伟 , 高峰 , 罗雄麟 . 基于值函数和策略梯度的深度强化学习综述 [J ] . 计算机学报 , 2019 , 42 ( 6 ): 1406 - 1438 .

LIU J W , GAO F , LUO X L . Survey of deep reinforcement learning based on value function and policy gradient [J ] . Chinese Journal of Computers , 2019 , 42 ( 6 ): 1406 - 1438 .

SUTTON R S . Learning to predict by the methods of temporal differences [J ] . Machine Learning , 1988 , 3 ( 1 ): 9 - 44 .

WATKINS C J C H , DAYAN P . Q-learning [J ] . Machine Learning , 1992 , 8 ( 3-4 ): 279 - 292 .

VAN HASSELT H , GUEZ A , SILVER D , et al . Deep reinforcement learning with double Q-learning [C ] // The 30th AAAI Conference on Artificial Intelligence .[S.l.:s.n. ] , 2016 .

SCHAUL T , QUAN J , ANTONOGLOU I , et al . Prioritized experience replay [C ] // The 4th International Conference on Learning Representations .[S.l.:s.n. ] , 2016 .

WANG Z , SCHAUL T , HESSEL M , et al . Dueling network architectures for deep reinforcement learning [C ] // The 33rd International Conference on Machine Learning . New York:ACM Press , 2016 .

NAIR A , SRINIVASAN P , BLACKWELL S , et al . Massively parallel methods for deep reinforcement learning [J ] . arXiv preprint , 2015 ,arXiv:1507. 04296

SLIVER D , LEVER G , HEESS N , et al . Deterministic policy gradient algorithms [C ] // The 31st International Conference on Machine Learning . New York:ACM Press , 2014 .

LILLICRAP P T , HUNT J J , PRITZEL A , et al . Continuous control with deep reinforcement learning [C ] // The 4th International Conference on Learning Representations .[S.l.:s.n. ] , 2016 .

MNIH V , BADIA A P , MIRZA M , et al . Asynchronous methods for deep reinforcement learning [C ] // The 33rd International Conference on Machine Learning . New York:ACM Press , 2016 .

SCHULMAN J , WOLSKI F , DHARIWAL P , et al . Proximal policy optimization algorithms [J ] . arXiv preprint , 2017 ,arXiv:1707. 06347 .

HAARNOJA T , ZHOU A , ABBEEL P , et al . Soft actor-critic:off-policy maximum entropy deep reinforcement learning with a stochastic actor [J ] . arXiv preprint , 2018 ,arXiv:1801. 01290 .

沈宇 , 韩金朋 , 李灵犀 , 等 . 游戏智能中的 AI——从多角色博弈到平行博弈 [J ] . 智能科学与技术学报 , 2020 , 2 ( 3 ): 205 - 213 .

SHEN Y , HAN J P , LI L X , et al . AI in game intelligence—from multi-role game to parallel game [J ] . Chinese Journal of Intelligent Science and Technology , 2020 , 2 ( 3 ): 205 - 213 .

BADIA A P , PIOT B , KAPTUROWSKI S , et al . Agent57:outperforming the atari human benchmark [J ] . arXiv preprint , 2020 ,arXiv:2003. 13350 .

KEMPKA M , WYDMUCH M , RUNC G , et al . Vizdoom:a doom-based AI research platform for visual reinforcement learning [C ] // 2016 IEEE Conference on Computational Intelligence and Games (CIG) . Piscataway:IEEE Press , 2016 : 1 - 8 .

LAMPLE G , CHAPLOT D S . Playing FPS games with deep reinforcement learning [C ] // The 31st AAAI Conference on Artificial Intelligence .[S.l.:s.n. ] , 2017 .

DOSOVITSKIY A , KOLTUN V . Learning to act by predicting the future [J ] . arXiv preprint , 2016 ,arXiv:1611. 01779 .

PATHAK D , AGRAWAL P , EFROS A A , et al . Curiosity-driven exploration by self-supervised prediction [C ] // The 34th International Conference on Machine Learning . New York:ACM Press , 2017 .

WU Y , ZHANG W , SONG K . Master-slave curriculum design for reinforcement learning [C ] // The 28th International Joint Conference on Artificial Intelligence . New York:ACM Press , 2018 : 1523 - 1529 .

VINYALS O , EWALDS T , BARTUNOV S , et al . StarcraftII:a new challenge for reinforcement learning [J ] . arXiv preprint , 2017 ,arXiv:1708. 04782 .

ZAMBALDI V , RAPOSO D , SANTORO A , et al . Relational deep reinforcement learning [J ] . arXiv preprint , 2018 ,arXiv:1806. 01830 .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [C ] // Advances in Neural Information Processing Systems . New York:ACM Press , 2017 : 5998 - 6008 .

RASHID T , SAMVELYAN M , DE WITT C S , et al . QMIX:monotonic value function factorisation for deep multi-agent reinforcement learning [J ] . arXiv preprint , 2018 ,arXiv:1803. 11485 .

YE D , LIU Z , SUN M , et al . Mastering complex control in MOBA games with deep reinforcement learning [C ] // The 34th AAAI Conference on Artificial Intelligence .[S.l.:s.n. ] , 2020 : 6672 - 6679 .

OH J , CHOCKALINGAM V , SINGH S , et al . Control of memory,active perception,and action in minecraft [C ] // The 33rd International Conference on Machine Learning . New York:ACM Press , 2016 .

JADERBERG M , MNIH V , CZARNECKI W M , et al . Reinforcement learning with unsupervised auxiliary tasks [J ] . arXiv preprint , 2016 ,arXiv:1611. 05397 .

MIROWSKI P , PASCANU R , VIOLA F , et al . Learning to navigate in complex environments [J ] . arXiv preprint , 2016 ,arXiv:1611. 03673 .

WANG Y , HE H , SUN C . Learning to navigate through complex dynamic environment with modular deep reinforcement learning [J ] . IEEE Transactions on Games , 2018 , 10 ( 4 ): 400 - 412 .

SHI H , SHI L , XU M , et al . End-to-end navigation strategy with deep reinforcement learning for mobile robots [J ] . IEEE Transactions on Industrial Informatics , 2020 , 16 ( 4 ): 2393 - 2402 .

SAVINOV N , RAICHUK A , MARINIER R , et al . Episodic curiosity through reachability [C ] // The 7th International Conference on Learning Representations .[S.l.:s.n. ] , 2019 .

ZHU Y , MOTTAGHI R , KOLVE E , et al . Target-driven visual navigation in indoor scenes using deep reinforcement learning [C ] // 2017 IEEE international conference on robotics and automation (ICRA) . Piscataway:IEEE Press , 2017 : 3357 - 3364 .

TAI L , LIU M . Towards cognitive exploration through deep reinforcement learning for mobile robots [J ] . arXiv preprint , 2016 ,arXiv:1610. 01733 .

TAI L , PAOLO G , LIU M . Virtual-to-real deep reinforcement learning:continuous control of mobile robots for mapless navigation [C ] // 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . Piscataway:IEEE Press , 2017 : 31 - 36 .

WU Y , RAO Z , ZHANG W , et al . Exploring the task cooperation in multi-goal visual navigation [C ] // The 28th International Joint Conference on Artificial Intelligence .[S.l.:s.n. ] , 2019 : 609 - 615 .

ZHANG W , ZHANG Y , LIU N . Map-less navigation:a single DRL-based controller for robots with varied dimensions [J ] . arXiv preprint , 2020 ,arXiv:2002. 06320 .

MIROWSKI P , GRIMES M K , MALINOWSKI M , et al . Learning to navigate in cities without a map [C ] // Advances in Neural Information Processing Systems .[S.l.:s.n . ] , 2018 : 2419 - 2430 .

LI A , HU H , MIROWSKI P , et al . Cross-view policy learning for street navigation [C ] // The IEEE International Conference on Computer Vision . Piscataway:IEEE Press , 2019 : 8100 - 8109 .

HERMANN K M , MALINOWSKI M , MIROWSKI P , et al . Learning to follow directions in street view [C ] // The 34th AAAI Conference on Artificial Intelligence .[S.l.:s.n. ] , 2020 .

CHANCÁ N M , MILFORD M . CityLearn:diverse real-world environments for sample-efficient navigation policy learning [J ] . arXiv preprint , 2020 ,arXiv:1910. 04335 .

孙长银 , 穆朝絮 . 多智能体深度强化学习的若干关键科学问题 [J ] . 自动化学报 , 2020 , 46 ( 7 ): 1301 - 1312 .

SUN C Y , MU C X . Important scientific problems of multi-agentdeep reinforcement learning [J ] . Acta Automatica Sinica , 2020 , 46 ( 7 ): 1301 - 1312 .

OROOJLOOYJADID A , HAJINEZHAD D . A review of cooperative multi-agent deep reinforcement learning [J ] . arXiv preprint , 2019 ,arXiv:1908. 03963 .

OMIDSHAFIEI S , PAZIS J , AMATO C , et al . Deep decentralized multi-task multi-agent reinforcement learning under partial observability [C ] // The 34th International Conference on Machine Learning . New York:ACM Press , 2017 .

MATIGNON L , LAURENT G J , LE FORT-PIAT N . Hysteretic Q-learning:an algorithm for decentralized reinforcement learning in cooperative multi-agent teams [C ] // 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2007 : 64 - 69 .

FOERSTER J , NARDELLI N , FARQUHAR G , et al . Stabilising experience replay for deep multi-agent reinforcement learning [C ] // The 34th International Conference on Machine Learning . New York:ACM Press , 2017 .

PALMER G , TUYLS K , BLOEMBERGEN D , et al . Lenient multi-agent deep reinforcement learning [C ] // The 17th International Conference on Autonomous agents and Multiagent Systems . New York:ACM Press , 2018 .

EVERETT R , ROBERTS S . Learning against non-stationary agents with opponent modelling and deep reinforcement learning [C ] // 2018 AAAI Spring Symposium Series .[S.l.:s.n. ] , 2018 .

JIN Y , WEI S , YUAN J , et al . Stabilizing multi-agent deep reinforcement learning by implicitly estimating other agents’ behaviors [C ] // 2020 IEEE International Conference on Acoustics,Speech and Signal Processing . Piscataway:IEEE Press , 2020 : 3547 - 3551 .

LIU X , TAN Y . Attentive relational state representation in decentralized multiagent reinforcement learning [J ] . IEEE Transactions on Cybernetics , 2020 .

GUPTA J K , EGOROV M , KOCHENDERFER M . Cooperative multi-agent control using deep reinforcement learning [C ] // The 16th International Conference on Autonomous Agents and Multiagent Systems . Cham:Springer , 2017 : 66 - 83 .

LOWE R , WU Y , TAMAR A , et al . Multi-agent actor-critic for mixed cooperative-competitive environments [C ] // Advances in Neural Information Processing Systems . New York:ACM Press , 2017 : 6379 - 6390 .

FOERSTER J , FARQUHAR G , AFOURAS T , et al . Counterfactual multi-agent policy gradients [C ] // The 32nd AAAI Conferenceon Artificial Intelligence .[S.l.:s.n. ] , 2018 .

SUNEHAG P , LEVER G , GRUSLYS A , et al . Value-decomposition networks for cooperative multi-agent learning [J ] . arXiv preprint , 2011 ,arXiv:1706. 05296 .

MAO H , ZHANG Z , XIAO Z , et al . Modelling the dynamic joint policy of teammates with attention multi-agent DDPG [C ] // The 18th International Conference on Autonomous Agentsand Multiagent Systems . New York:ACM Press , 2019 .

IQBAL S , SHA F . Actor-attention-critic for multi-agent reinforcement learning [C ] // International Conference on Machine Learning .[S.l.:s.n. ] , 2019 : 2961 - 2970 .

FOERSTER J N , ASSAEL Y M , DE FREITAS N , et al . Learning to communicate with deep multi-agent reinforcement learning [C ] // Advances in Neural Information Processing Systems . New York:ACM Press , 2016 : 2137 - 2145 .

SUKHBAATAR S , SZLAM A , FERGUS R . Learning multiagent communication with back propagation [C ] // Advances in Neural Information Processing Systems . New York:ACM Press , 2016 : 2244 - 2252 .

JIANG J , LU Z . Learning attentional communication for multi-agent cooperation [C ] // Advances in Neural Information Processing Systems . New York:ACM Press , 2018 : 7254 - 7264 .

KIM D , MOON S , HOSTALLERO D , et al . Learning to schedule communication in multi-agent reinforcement learning [C ] // The 7th International Conference on Learning Representations .[S.l.:s.n. ] , 2019 .

DAS A , GERVET T , ROMOFF J , et al . TarMAC:targeted multi-agent communication [C ] // The 36th International Conference on MachineLearning .[S.l.:s.n. ] , 2019 .

SHANI G , HECKERMAN D , BRAFMAN R I , et al . An MDP-based recommender system [J ] . Journal of Machine Learning Research , 2005 , 6 ( Sep ): 1265 - 1295 .

ZHAO X , XIA L , TANG J , et al . Deep reinforcement learning for search,recommendation,and online advertising:a survey [J ] . ACM SIGWEB Newsletter , 2019 ( Spring ): 1 - 15 .

ZHAO X , XIA L , ZHANG L , et al . Deep reinforcement learning for page-wise recommendations [C ] // The 12th ACM Conference on Recommender Systems . New York:ACM Press , 2018 : 95 - 103 .

ZHENG G , ZHANG F , ZHENG Z , et al . DRN:a deep reinforcement learning framework for news recommendation [C ] // The 2018 World Wide Web Conference . New York:ACM Press , 2018 : 167 - 176 .

Views

270

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

AI in game intelligence—from multi-role game to parallel game

Digital chefs and intelligent cooking systems based on multimodal large language model

New AI for new frontier of intelligent science and technology: from PAPAI++ to new enterprise ecology

Review: the application of deep reinforcement learning to quantitative trading in financial market

K21 education and intelligent sciences: future schools and teachers/students in future society

Related Author

Yu SHEN

Jinpeng HAN

Lingxi LI

Fei-Yue WANG

LI Xinyuan

LI Bai

SUN Yueshuo

ZHANG Tantan

Related Institution

School of Artificial Intelligence, University of Chinese Academy of Sciences

The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences

School of Software Engineering,Xi’an Jiaotong University

Department of Electrical and Computer Engineering,Indiana University-Purdue University Indianapolis

Qingdao Academy of Intelligent Industries

Postal code：100079
Tel：（010）53879206 Email：cjist@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：35685 Visits Today：160

⁰