A cooperative multi-agent reinforcement learning algorithm based on dynamic self-selection parameters sharing

Han WANG; Yang YU; Yuan JIANG

doi:10.11959/j.issn.2096-6652.202214

您当前的位置：

首页 >

文章列表页 >

A cooperative multi-agent reinforcement learning algorithm based on dynamic self-selection parameters sharing

Special Topic: Crowd Intelligence | 更新时间：2024-06-05

- A cooperative multi-agent reinforcement learning algorithm based on dynamic self-selection parameters sharing
- Chinese Journal of Intelligent Science and Technology Vol. 4, Issue 1, Pages: 75-83(2022)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-6652.202214
  CLC： TP181
- Published Online：2022-03，
  
  Published：15 March 2022
- 稿件说明：
移动端阅览
Han WANG, Yang YU, Yuan JIANG. A cooperative multi-agent reinforcement learning algorithm based on dynamic self-selection parameters sharing[J]. Chinese journal of intelligent science and technology, 2022, 4(1): 75-83.
DOI：

Han WANG, Yang YU, Yuan JIANG. A cooperative multi-agent reinforcement learning algorithm based on dynamic self-selection parameters sharing[J]. Chinese journal of intelligent science and technology, 2022, 4(1): 75-83. DOI： 10.11959/j.issn.2096-6652.202214.

摘要

在多智能体强化学习的研究中，参数共享作为学习过程中一种信息集中的方式，可以有效地缓解不稳定性导致的学习低效性。但是，在实际应用中智能体使用同样的策略往往会带来不利影响。为了解决此类过度共享的问题，提出了一种新的方法来赋予智能体自动识别可能受益于共享参数的智能体的能力，并且可以在学习过程中动态地选择共享参数的对象。具体来说，智能体需要将历史轨迹编码为可表示其潜在意图的隐信息，并通过与其余智能体隐信息的对比选择共享参数的对象。实验表明，提出的方法在多智能体系统中不仅可以提高参数共享的效率，同时保证了策略学习的质量。

Abstract

In multi-agent reinforcement learning

parameter sharing can effectively alleviate the inefficiency of learning caused by non-stationarity.However

maintaining the same policy forall agents during learning may have detrimental effects.To solve this problem

a new approach was introduced to give agents the ability to automatically identify agents that may benefit from parameter sharing and dynamically share parameters them during learning.Specifically

agents needed to encode empirical trajectories as implicit information that can represent their potential intentions

and selected peers to share parameters by comparing their intentions.Experiments show that the proposed method not only can improve the efficiency of parameter sharing

but also ensure the quality of policy learning in multi-agent system.

关键词

Keywords

references

刘全 , 翟建伟 , 章宗长 , 等 . 深度强化学习综述 [J ] . 计算机学报 , 2018 , 41 ( 1 ): 1 - 27 .

LIU Q , ZHAI J W , ZHANG Z Z , et al . A survey on deep reinforcement learning [J ] . Chinese Journal of Computers , 2018 , 41 ( 1 ): 1 - 27 .

王飞跃 , 曹东璞 , 魏庆来 . 强化学习:迈向知行合一的智能机制与算法 [J ] . 智能科学与技术学报 , 2020 , 2 ( 2 ): 101 - 106 .

WANG F Y , CAO D P , WEI Q L . Reinforcement learning:toward action-knowledge merged intelligent mechanisms and algorithms [J ] . Chinese Journal of Intelligent Science and Technology , 2020 , 2 ( 2 ): 101 - 106 .

刘朝阳 , 穆朝絮 , 孙长银 . 深度强化学习算法与应用研究现状综述 [J ] . 智能科学与技术学报 , 2020 , 2 ( 4 ): 314 - 326 .

LIU Z Y , MU C X , SUN C Y . An overview on algorithms and applications of deep reinforcement learning [J ] . Chinese Journal of Intelligent Science and Technology , 2020 , 2 ( 4 ): 314 - 326 .

王金予 , 魏欣然 , 石文磊 , 等 . 强化学习在资源优化领域的应用 [J ] . 大数据 , 2021 , 7 ( 5 ): 131 - 149 .

WANG J Y , WEI X R , SHI W L , et al . Applications of reinforcement learning in the field of resource optimization [J ] . Big Data Research , 2021 , 7 ( 5 ): 131 - 149 .

LOWE R , WU Y , TAMAR A , et al . Multi-agent actor-critic for mixed cooperative-competitive environments [C ] // Proceedings of the 31st Annual Conference on Neural Information Processing Systems . Cambridge:MIT Press , 2017 .

CONITZER V , SANDHOLM T . AWESOME:a general multiagent learning algorithm that converges in self-play and learns a best response against stationary opponents [J ] . Machine Learning , 2007 , 67 ( 1/2 ): 23 - 43 .

EVERETT R , ROBERTS S . Learning against non-stationary agentswith opponent modelling and deep reinforcement learning [C ] // Proceedings of AAAI Spring Symposium Series .[S.l.:s.n. ] , 2018 .

FOERSTER J N , ASSAEL Y M , DE FREITAS N , et al . Learning to communicate with deep multi-agent reinforcement learning [C ] // Proceedings of Advances in Neural Information Processing Systems . New York:ACM Press , 2016 : 2137 - 2145 .

SUKHBAATAR S , SZLAM A , FERGUS R . Learning multiagentcommunication with back propagation [C ] // Proceedings of Advances in Neural Information Processing Systems . New York:ACM Press , 2016 : 2244 - 2252 .

KIM D , MOON S , HOSTALLERO D , et al . Learning to schedule communication in multi-agent reinforcement learning [C ] // Proceedings of the 7th International Conference on Learning Representations .[S.l.:s.n. ] , 2019 .

SUNEHAG P , LEVER G , GRUSLYS A , et al . Value-decomposition networks for cooperative multi-agent learning [J ] . arXiv preprint , 2017 ,arXiv:1706.05296.

RASHID T , SAMVELYAN M , WITT C S D , et al . QMIX:monotonic value function factorisation for deep multi-agent reinforcement learning [C ] // Proceedings of the 35th International Conference on Machine Learning . New York:JMLR , 2018 : 4292 - 4301 .

GUPTA J K , EGOROV M , KOCHENDERFER M J . Cooperative multi-agent control using deep reinforcement learning [C ] // Proceedings of the 18th International Conference on Autonomous Agents and Multiagent Systems . New York:ACM Press , 2017 : 66 - 83 .

BAI H Y , CAI S J , YE N , et al . Intention-aware online POMDP planning for autonomous driving in a crowd [C ] // Proceedings of 2015 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2015 : 454 - 460 .

SADIGH D , SASTRY S S , SESHIA S A , et al . Information gathering actions over human internal state [C ] // Proceedings of 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . Piscataway:IEEE Press , 2016 : 66 - 73 .

XIE A N , LOSEY D P , TOLSMA R , et al . Learning latent representations to influence multi-agent interaction [C ] // Proceedings of the 4th Conference on Robot Learning . New York:JMLR , 2020 : 575 - 588 .

LITTMAN M L . Markov games as a framework for multi-agent reinforcement learning [M ] // Machine learning proceedings 1994 . Amsterdam : Elsevier , 1994 : 157 - 163 .

TERRY J K , GRAMMEL N , HARI A , et al . Revisiting parameter sharing in multi-agent deep reinforcement learning [J ] . arXiv preprint , 2020 ,arXiv:2005.13625.

CHRISTIANOS F , PAPOUDAKIS G , RAHMAN A , et al . Scaling multi-agent reinforcement learning with selective parameter sharing [C ] // Proceedings of the 38th International Conference on Machine Learning . New York:JMLR , 2021 : 1989 - 1998 .

QI S Y , ZHU S C . Intent-aware multi-agent reinforcement learning [C ] // Proceedings of 2018 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2018 : 7533 - 7540 .

KIM W J , PARK J G , SUNG T C . Communication in multi-agent reinforcement learning:intention sharing [C ] // Proceedings of the 9th International Conference on Learning Representations .[S.l.:s.n. ] , 2021 .

SAMVELYAN M , RASHID T , WITT C S , et al . The StarCraft multi-agent challenge [C ] // Proceedings of the 18th International Conference on Autonomous Agents and Multi-Agent Systems . New York:ACM Press , 2019 : 2186 - 2188 .

Views

1291

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Output synchronization of heterogeneous multi-agent system:a reinforcement learning approach based on data

An overview of optimal consensus for data driven multi-agent system based on reinforcement learning

A path planning method for complex naval battle field based on an improved DQN algorithm

Research development on automated robotic peg-in-hole assembly

Synchronization control of unknown heterogeneous multi-agent system via model-free adaptive dynamic programming

Related Author

Zhanshan WANG

Yingying LIU

Jinna LI

Weiran CHENG

Haitao YUAN

Jing BI

Zhou YU

Fangbo QIN

Related Institution

Faculty of Information Technology, Beijing University of Technology

School of Automation Science and Electrical Engineering, Beihang University

School of Automation and Electrical Engineering, University of Science and Technology Beijing

Research Institute of Urbanization and Urban Safety, University of Science and Technology Beijing

Center for Intelligent and Networked Systems, Department of Automation, Tsinghua University

Postal code：100079
Tel：（010）53879206 Email：cjist@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：35685 Visits Today：160

⁰