面向低轨星座边缘计算的博弈强化学习方法综述

谷学强; 张万鹏; 谭思雨; 罗俊仁; 周棪忠

doi:10.11959/j.issn.2096-6652.202432

您当前的位置：

首页 >

文章列表页 >

面向低轨星座边缘计算的博弈强化学习方法综述

综述与展望 | 更新时间：2024-11-04

- 面向低轨星座边缘计算的博弈强化学习方法综述
- Overview on game reinforcement learning methods for edge computing of low-orbit constellation
- 智能科学与技术学报 2024年6卷第3期页码：301-318
- 作者机构：
  
  1.国防科技大学智能科学学院，湖南长沙 410073
  2.湖南先进技术研究院，湖南长沙 410205
- 作者简介：
  
  [ "谷学强（1983- ），男，博士，国防科技大学智能科学学院副研究员，主要研究方向为智能规划与决策、边缘智能等。" ]
  [ "张万鹏（1981- ），男，博士，国防科技大学智能科学学院研究员，博士生导师，主要研究方向为大数据智能、智能演进等。" ]
  [ "谭思雨（1996- ），女，湖南先进技术研究院算法工程师，主要研究方向为深度学习与影像智能处理。" ]
  [ "罗俊仁（1989- ），男，国防科技大学智能科学学院博士生，主要研究方向为不完美信息博弈、多智能体学习等。" ]
  [ "周棪忠（2000- ），男，国防科技大学智能科学学院硕士研究生，主要研究方向为大模型规划、智能演进。" ]
- 基金信息：
  
  国家自然科学基金项目(92271108;62173336)
- DOI：10.11959/j.issn.2096-6652.202432
  中图分类号： TP181
- 收稿：2024-05-22，
  
  修回：2024-07-25，
  
  纸质出版：2024-09-15
- 稿件说明：
移动端阅览
谷学强,张万鹏,谭思雨等.面向低轨星座边缘计算的博弈强化学习方法综述[J].智能科学与技术学报,2024,06(03):301-318.

GU Xueqiang,ZHANG Wanpeng,TAN Siyu,et al.Overview on game reinforcement learning methods for edge computing of low-orbit constellation[J].Chinese Journal of Intelligent Science and Technology,2024,06(03):301-318.
谷学强,张万鹏,谭思雨等.面向低轨星座边缘计算的博弈强化学习方法综述[J].智能科学与技术学报,2024,06(03):301-318. DOI： 10.11959/j.issn.2096-6652.202432.

GU Xueqiang,ZHANG Wanpeng,TAN Siyu,et al.Overview on game reinforcement learning methods for edge computing of low-orbit constellation[J].Chinese Journal of Intelligent Science and Technology,2024,06(03):301-318. DOI： 10.11959/j.issn.2096-6652.202432.

摘要

博弈强化学习作为人工智能领域的新兴范式，是当前解决低轨星座边缘计算问题的主流方法。融入博弈论的多智能体深度强化学习方法为复杂、动态、不确定性的星座边缘计算问题提供了新思路。通过梳理总结卫星组网、任务卸载以及资源调度3种卫星边缘计算主要研究方向，详细阐述了博弈强化学习范式基础，并从博弈模型、深度Q网络、深度确定性策略梯度以及近端策略优化等方面分别阐述了3种研究方向上的典型应用现状，最后对该领域的前沿挑战进行分析，期望为博弈强化学习范式与低轨星座边缘计算领域的交叉融合研究提供参考。

Abstract

As a new paradigm in the field of artificial intelligence

game reinforcement learning is an advanced mainstream method to solve the edge computing problem of low-orbit constellation. The multi-agent deep reinforcement learning integrated into the game perspective provides a new idea for dynamic

complex and uncertain constellation edge computing problems. By summarizing the three main research directions of satellite edge computing

namely satellite networking

task unloading and resource scheduling

the basis of game reinforcement learning paradigm is elaborated

and the typical applications in the three research directions are described respectively from the methods of game model

deep Q network

deep deterministic strategy gradient and near-end strategy optimization. In the end

the paper looks forward to the frontier challenges in this field

expected to provide a reference for the cross-fusion research of game reinforcement learning paradigm and low-orbit constellation edge computing.

关键词

Keywords

references

ZHI F K, SHI S H, YU L L, et al. Research on the development status and key technologies of large LEO remote sensing constellations[J]. Spacecraft Recovery & Remote Sensing, 2023, 44(1): 93-101.

LEE J Y, LIM B, KO Y C. Performance analysis of multi-hop low earth orbit satellite network over mixed RF/FSO links[J]. ICT Express, 2024.

于帆, 垣信. 卫星融资67亿元布局低轨卫星互联网[N].中国信息化周报, 2024-02-26(011).

YU F. YUAN X. Satellite raised 6.7 billion yuan to lay out the low-orbit satellite Internet[N]. China Information Weekly, 2024-02-26(011).

LIN X Q, CIONI S, CHARBIT G, et al. On the path to 6G: embracing the next wave of low earth orbit satellite access[J]. IEEE Communications Magazine, 2021, 59(12): 36-42.

JIANG W W. Software defined satellite networks: a survey[J]. Digital Communications and Networks, 2023, 9(6): 1243-1264.

WANG B, FENG T, HUANG D Y. A joint computation offloading and resource allocation strategy for LEO satellite edge computing system[C]//Proceedings of the 2020 IEEE 20th International Conference on Communication Technology (ICCT). Piscataway: IEEE Press, 2020: 649-655.

CUI J J, LIU Y W, NALLANATHAN A. Multi-agent reinforcement learning-based resource allocation for UAV networks[J]. IEEE Transactions on Wireless Communications, 2020, 19(2): 729-743.

SALAS M, FERNÁNDEZ J, VAN DEN IJSSEL J. Precise relative navigation in medium earth orbits with global navigation satellite systems and intersatellite links for black hole imaging[J]. Advances in Space Research, 2024, 74(6): 2599-2614.

朱振才, 张晟宇, 胡海鹰. 分布式空间系统组网技术综述[J]. 航天器工程, 2022, 27(6): 77-83.

ZHU Z C, ZHANG S Y, HU H Y. Overview of networking technology of distributed space system[J]. Spacecraft engineering, 2022, 27(6): 77-83.

JIAN W, MIN J, LIANG Z. Research on 6G-oriented LEO satellite edge computing architecture[J]. Application of Electronic Technique, 2022, 48(6): 5-9, 13.

ZHUANG Z L, LI Y, SUN Y N, et al. Network-based dynamic dispatching rule generation mechanism for real-time production scheduling problems with dynamic job arrivals[J]. Robotics and Computer-Integrated Manufacturing, 2022, 73: 102261.

PROL F S, FERRE R M, SALEEM Z, et al. Position, navigation, and timing (PNT) through low earth orbit (LEO) satellites: a survey on current status, challenges, and opportunities[J]. IEEE Access, 2022, 10: 83971-84002.

CASSARÁ P, GOTTA A, MARCHESE M, et al. Orbital edge offloading on mega-LEO satellite constellations for equal access to computing[J]. IEEE Communications Magazine, 2022, 60(4): 32-36.

ABDELSADEK M Y, CHAUDHRY A U, DARWISH T, et al. Future space networks: toward the next giant leap for humankind[J]. IEEE Transactions on Communications, 2023, 71(2): 949-1007.

LIN L, LIAO X F, JIN H, et al. Computation offloading toward edge computing[J]. Proceedings of the IEEE, 2019, 107(8): 1584-1607.

SADATDIYNOV K, CUI L Z, ZHANG L, et al. A review of optimization methods for computation offloading in edge computing networks[J]. Digital Communications and Networks, 2023, 9(2): 450-461.

周绪, 苗辉, 杨静, 等. 边缘计算资源调度:历史、架构、建模与方法分析[J].计算机集成制造系统, 已录用, doi: 10.13196/j.cims.2023.0132.

ZHOU X, MIAO H, YANG J, et al. Edge computing resource scheduling: History, architecture, modeling and Method analysis [J]. Department of Computer Integrated Manufacturing, 2024: accepted, doi: 10.13196/j.cims.2023.0132.

陈全, 杨磊, 郭剑鸣, 等.低轨巨型星座网络: 组网技术与研究现状[J]. 通信学报, 2022, 43(5): 177-189.

CHEN Q, YANG L, GUO J M, et al. Low orbit giant constellation network: Networking technology and research status [J]. Journal of Communication, 2022, 43(5): 177-189.

王鹏, 张佳鑫, 张兴, 等. 低轨卫星智能多接入边缘计算网络: 需求、架构、机遇与挑战[J]. 移动通信, 2021, 45(5): 35-46.

WANG P, ZHANG J X, ZHANG X, et al. Low earth orbit satellite intelligent multi-access edge computing networks: requirements, architecture, opportunities and challenges[J]. Mobile Communications, 2021, 45(5): 35-46.

高媛, 方海, 赵扬, 等. 星地协同网络中的边缘计算技术综述[J]. 空间电子技术, 2023, 20(2): 15-21.

GAO Y, FANG H, ZHAO Y, et al. A survey on edge computing in satellite-terrestrial networks[J]. Space Electronic Technology, 2023, 20(2): 15-21.

吴戌, 秦大国, 李阳, 等. 遥感卫星边缘计算架构设计与关键技术分析[J].信息工程大学学报, 2023, 24(3): 336-342.

WU X, QIN D G, LI Y, et al. Remote sensing satellite edge computing architecture design and key technology analysis [J]. Journal of Information Engineering University, 2023, 24(3): 336-342.

KALOEV M, KRASTEV G. Experiments focused on exploration in deep reinforcement learning[C]//Proceedings of the 2021 5th International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT). Piscataway: IEEE Press, 2021: 351-355.

KALOEV M, KRASTEV G. Comprehensive review of benefits from the use of sparse updates techniques in reinforcement learning: experimental simulations in complex action space environments[C]//Proceedings of the 2023 4th International Conference on Communications, Information, Electronic and Energy Systems (CIEES). Piscataway: IEEE Press, 2023: 1-7.

LIU W, ZHANG S C, ZHANG X P, et al. Distributed game coordination control method based on reinforcement learning for multi-microgrid[J]. Energy Reports, 2023, 9: 912-921.

KAZEMI N, AZGHANI M. Secure spectrum sharing and power allocation by multi agent reinforcement learning[J]. Digital Signal Processing, 2024, 146: 104369.

ZHU Y Y, YIN X, CHEN C L. Extracting decision tree from trained deep reinforcement learning in traffic signal control[J]. IEEE Transactions on Computational Social Systems, 2023, 10(4): 1997-2007.

ANTHONY T, ECCLES T, TACCHETTI A, et al. Learning to play No-press diplomacy with best response policy iteration[EB ] . arXiv preprint, 2020, arXiv: 2006.04635 .

JIANG C L, CAO T F, GUAN J F. Intelligent task offloading and collaborative computation over D2D communication[J]. China Communications, 2021, 18(3): 251-263.

HONG Z C, CHEN W H, HUANG H W, et al. Multi-hop cooperative computation offloading for industrial IoT-edge-cloud computing environments[J]. IEEE Transactions on Parallel and Distributed Systems, 2019, 30(12): 2759-2774.

TRAN T X, POMPILI D. Joint task offloading and resource allocation for multi-server mobile-edge computing networks[J]. IEEE Transactions on Vehicular Technology, 2019, 68(1): 856-868.

DAO M, LIU Z, GUO S, et al. A computation offloading and resource allocation mechanism based on minimizing devices energy consumption and system delay[J]. Journal of Electronics & Information Technology, 2019, 41(11): 2684-2690.

滕颖蕾, 刘薇, 欧阳卫平, 等. 无线MEC系统中队列状态感知的卸载和传输联合优化[J]. 北京邮电大学学报, 2019, 42(3): 14-20.

TENG Y L, LIU W, OUYANG W P, et al. Joint offloading and transmission optimization of queue state awareness in wireless MEC systems[J]. Journal of Beijing University of Posts and Telecommunications, 2019, 42(3): 14-20.

邓志龙, 张琦玮, 曹皓, 等. 一种基于深度强化学习的调度优化方法[J]. 西北工业大学学报, 2017, 35(6): 1047-1053.

DENG Z L, ZHANG Q W, CAO H, et al. A scheduling optimization method based on depth intensive study[J]. Journal of Northwestern Polytechnical University, 2017, 35(6): 1047-1053.

CUI Y P, LIANG Y J, WANG R Y. Resource allocation algorithm with multi-platform intelligent offloading in D2D-enabled vehicular networks[J]. IEEE Access, 2019, 7: 21246-21253.

MENG H, CHAO D C, GUO Q Y, et al. Delay-sensitive task scheduling with deep reinforcement learning in mobile-edge computing systems[J]. Journal of Physics: Conference Series, 2019, 1229(1): 012059.

HU Z B, WANG S Z, HU L S, et al. Optimization of task unloading strategy based on Game Theory in cloud edge collaborative system[C]//Proceedings of the 8th International Conference on Computing and Artificial Intelligence. New York: ACM, 2022: 228-234.

DARWISH T, KURT G K, YANIKOMEROGLU H, et al. LEO satellites in 5G and beyond networks: a review from a standardization perspective[J]. IEEE Access, 2022, 10: 35040-35060.

XIAO Z Y, YANG J Y, MAO T Q, et al. LEO satellite access network (LEO-SAN) toward 6G: challenges and approaches[J]. IEEE Wireless Communications, 2024, 31(2): 89-96.

JI B, CAO X Y, YANG K. Research and simulation of satellite networking technology based on link detection mechanism[J]. Electronic Information Warfare Technology, 2022, 37(1): 90-96.

RAJAGOPAL A, RAMACHANDRAN A, SHANKAR K, et al. Optimal routing strategy based on extreme learning machine with beetle antennae search algorithm for Low Earth Orbit satellite communication networks[J]. International Journal of Satellite Communications and Networking, 2021, 39(3): 305-317.

OUYANG M, ZHANG R, WANG B Q, et al. Network coding-based multipath transmission for LEO satellite networks with domain cluster[J]. IEEE Internet of Things Journal, 2024, 11(12): 21659-21673.

FENG L, LIU Y F, WU L, et al. A satellite handover strategy based on MIMO technology in LEO satellite networks[J]. IEEE Communications Letters, 2020, 24(7): 1505-1509.

LI Y T, ZHOU W Y, ZHOU S L. Forecast based handover in an extensible multi-layer LEO mobile satellite system[J]. IEEE Access, 2020, 8: 42768-42783.

DU P P, LI J D, BAI W G, et al. Dual location area based distributed location management for hybrid LEO/MEO mega satellite networks[J]. IEEE Transactions on Vehicular Technology, 2023, 72(2): 2307-2321.

ZHU H T, GUO Q. User group based multi-satellite handover strategy for LEO satellite networks[J]. Telecommunications Science, 2022, 38(4): 39-48.

ROTH M, BRANDT H, BISCHL H. Implementation of a geographical routing scheme for low Earth orbiting satellite constellations using intersatellite links[J]. International Journal of Satellite Communications and Networking, 2021, 39(1): 92-107.

CHEN Q, CHEN X Q, YANG L, et al. A distributed congestion avoidance routing algorithm in mega-constellation network with multi-gateway[J]. Acta Astronautica, 2019, 162: 376-387.

WU S G, WANG H Y, WANG Y, et al. Research and analysis on routing technology of LEO satellite network[J]. Satellite & Network, 2021(9): 66-74.

ZHOU J, GONG X T, SUN L J, et al. Adaptive routing strategy based on improved double Q-learning for satellite Internet of Things[J]. Security and Communication Networks, 2021, 5530023.

倪少杰, 岳洋, 左勇, 等. 卫星网络路由技术现状及展望[J]. 电子与信息学报, 2023, 45(2): 383-395.

NI S J, YUE Y, ZUO Y, et al. The status quo and prospect of satellite network routing technology[J]. Journal of Electronics & Information Technology, 2023, 45(2): 383-395.

LIU Y, WANG L N, et al. LEO satellite intelligent-sensing routing algorithm based on a dendrite network[J]. Chinese Journal of Engineering, 2023, 45(3): 465-474.

ZHU D L, LIU H T, LI T, et al. Deep reinforcement learning-based task offloading in satellite-terrestrial edge computing networks[C]//Proceedings of the 2021 IEEE Wireless Communications and Networking Conference (WCNC). Piscataway: IEEE Press, 2021: 1-7.

LI C C, ZHANG Y S, XIE R C, et al. Integrating edge computing into low earth orbit satellite networks: architecture and prototype[J]. IEEE Access, 2021, 9: 39126-39137.

XIE R C, TANG Q Q, WANG Q N, et al. Satellite-terrestrial integrated edge computing networks: architecture, challenges, and open issues[J]. IEEE Network, 2020, 34(3): 224-231.

WANG J X, ZHANG F, WANG X M, et al. AN edge-to-edge collaborative approach for AI tasks in LEO networks[C]//Proceedings of the 2023 8th IEEE International Conference on Network Intelligence and Digital Content (IC-NIDC). Piscataway: IEEE Press, 2023: 300-304.

ABDERRAHIM W, AMIN O, ALOUINI M S, et al. Latency-aware offloading in integrated satellite terrestrial networks[J]. IEEE Open Journal of the Communications Society, 2020, 1: 490-500.

BI J, YUAN H T, DUANMU S F, et al. Energy-optimized partial computation offloading in mobile-edge computing with genetic simulated-annealing-based particle swarm optimization[J]. IEEE Internet of Things Journal, 2021, 8(5): 3774-3785.

TANG Q Q, FEI Z S, LI B, et al. Computation offloading in LEO satellite networks with hybrid cloud and edge computing[J]. IEEE Internet of Things Journal, 2021, 8(11): 9164-9176.

SHENG S. A resource collaborative scheduling strategy based on cloud edge framework[C]//Proceedings of the 2021 IEEE 5th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC). Piscataway: IEEE Press, 2021: 1740-1744.

GONG Y K, YAO H P, WANG J J, et al. Multi-agent driven resource allocation and interference management for deep edge networks[J]. IEEE Transactions on Vehicular Technology, 2022, 71(2): 2018-2030.

LI J L, SHANG Y, QIN M, et al. Multiobjective oriented task scheduling in heterogeneous mobile edge computing networks[J]. IEEE Transactions on Vehicular Technology, 2022, 71(8): 8955-8966.

AMINI MOTLAGH A, MOVAGHAR A, RAHMANI A M. Task scheduling mechanisms in cloud computing: a systematic review[J]. International Journal of Communication Systems, 2020, 33(6): e4302.

YUAN Y X, LEI L, VU T X, et al. Adapting to dynamic LEO-B5G systems: meta-critic learning based efficient resource scheduling[J]. IEEE Transactions on Wireless Communications, 2022, 21(11): 9582-9595.

YANG F, YANG S W, CHEN Y K, et al. Efficient pencil beam synthesis in 4-D antenna arrays using an iterative convex optimization algorithm[J]. IEEE Transactions on Antennas and Propagation, 2019, 67(11): 6847-6858.

LIU Z J, PANG S, PU C Y, et al. Research of reverse iterative control algorithm[C]//Proceedings of the 2018 Chinese Automation Congress (CAC). Piscataway: IEEE Press, 2018: 3428-3433.

LIU H, WANG K P, LIU Z K. Application and research of heuristic search algorithm in crawler field[C]//Proceedings of the 2020 2nd International Conference on Information Technology and Computer Application (ITCA). Piscataway: IEEE Press, 2020: 41-44.

NIU L W, CHEN X F, ZHANG N, et al. Multiagent meta-reinforcement learning for optimized task scheduling in heterogeneous edge computing systems[J]. IEEE Internet of Things Journal, 2023, 10(12): 10519-10531.

GENG L W, ZHAO H B, WANG J Y, et al. Deep-reinforcement-learning-based distributed computation offloading in vehicular edge computing networks[J]. IEEE Internet of Things Journal, 2023, 10(14): 12416-12433.

王尧, 罗俊仁, 周棪忠, 等. 面向策略探索的强化学习与进化计算方法综述[J]. 计算机科学, 2024, 51(3): 183-197.

WANG Y, LUO J R, ZHOU Y Z, et al. Review of reinforcement learning and evolutionary computation methods for strategy exploration[J]. Computer Science, 2024, 51(3): 183-197.

DANESHFAR F, BEVRANI H. Load-frequency control: a GA-based multi-agent reinforcement learning[J]. IET Generation, Transmission & Distribution, 2010, 4(1): 13.

ZOU Q J, JIANG Y J, GAO B, et al. An overview of cooperative multi-agent deep reinforcement learning[J]. Aero Weaponry, 2022, 29(6): 78-88.

LUO J R, ZHANG W P, XIANG F T, et al. Survey on intelligent wargaming: tactical & campaign wargame and strategic game from game-theoretic perspective[J]. Journal of System Simulation, 2023, 35(9): 1871-1894.

FAN W H, HUA M Y, ZHANG Y Y, et al. Game-based task offloading and resource allocation for vehicular edge computing with edge-edge cooperation[J]. IEEE Transactions on Vehicular Technology, 2023, 72(6): 7857-7870.

SAKIP A, YERSAINOV R, ATASHIKOVA M, et al. Lightweight energy-efficient offloading framework for mobile edge/cloud computing[C]//Proceedings of the 2023 17th International Conference on Ubiquitous Information Management and Communication (IMCOM). Piscataway: IEEE Press, 2023: 1-8.

LIANG Z Z, LIUT Y, HUANG K B, et al. I/O interference aware multiuser computation offloading for virtualized edge computing[C]//Proceedings of the 2019 IEEE International Conference on Communications (ICC). Piscataway: IEEE Press, 2019: 1-6.

ZHAO L R, DANG Z H, ZHANG Y L. Orbital game: concepts, principles and methods[J]. Journal of Command and Control, 2021, 7(3): 215-224.

XU X S, DANG Z H, SONG B, et al. Method for cluster satellite orbit pursuit-evasion game based on multi-agent deep deterministic policy gradient algorithm[J]. Aerospace Shanghai (Chinese & English), 2022, 39(2): 24-31.

PEDDI S B, PATIL S R. Game theory based vertical handoff decision model for media independent handover in heterogeneous wireless networks[C]//Proceedings of the 2016 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET), Piscataway: IEEE Press, 2016: 719-724.

CHARILAS D E, MARKAKI O I, VLACHEAS P T. Admission control as a non-cooperative multi-stage game between wireless networks[C]//Proceedings of the 2009 16th International Conference on Systems, Signals and Image Processing. Piscataway: IEEE Press, 2009: 1-5.

NIYATO D, HOSSAIN E. Dynamics of network selection in heterogeneous wireless networks: an evolutionary game approach[J]. IEEE Transactions on Vehicular Technology, 2009, 58(4): 2008-2017.

WANG H L, ZUO Y, CHANG Q. Target protection using multiple unmanned aerial vehicles based on Stackelberg security game[C]//Proceedings of the 2023 4th International Conference on Intelligent Design(ICID). Piscataway: IEEE Press, 2023: 169-174.

CHEN J X, WU Q H, XU Y H, et al. A multi-leader multi-follower stackelberg game for coalition-based UAV MEC networks[J]. IEEE Wireless Communications Letters, 2021, 10(11): 2350-2354.

底晓强, 于力伟, 刘旭, 等. 一种基于演化博弈的低轨卫星切换算法研究[J]. 南京大学学报, 2018, 54(4): 855-862.

DI X Q, YU L W, LIU X, et al. Research on a Low orbit satellite Switching Algorithm based on Evolutionary game[J]. Journal of Nanjing University, 2018, 54(4): 855-862.

ZHANG H M, LI Q, ZHANG Y J, et al. Game theory based power allocation method for inter-satellite links in LEO/MEO two-layered satellite networks[C]//Proceedings of the 2021 IEEE/CIC International Conference on Communications in China (ICCC). Piscataway: IEEE Press, 2021: 398- 403.

LI W, JIA L L, CHEN Q, et al. A game theory-based distributed downlink spectrum sharing method in large-scale hybrid satellite constellations[J]. IEEE Transactions on Communications, 2024, 72(8): 4620-4632.

冯彤. 面向低轨卫星的移动边缘计算卸载策略研究[D]. 桂林: 桂林电子科技大学, 2021.

FENG T. Research on unloading strategy of mobile edge computing for low-orbit satellites[D]. Guilin: Guilin University of Electronic Science and Technology, 2021.

FANG H, JIA Y Y, WANG Y L, et al. Matching game based task offloading and resource allocation algorithm for satellite edge computing networks[C]//Proceedings of the 2022 International Symposium on Networks, Computers and Communications (ISNCC). Piscataway: IEEE Press, 2022: 1-5.

谢杰成. 面向低轨卫星系统的多星边缘计算卸载策略研究[D]. 桂林: 桂林电子科技大学, 2023.

XIE J C. Research on multi-star edge computing unloading strategy for low-orbit satellite system [D]. Guilin: Guilin University of Electronic Science and Technology, 2023.

TONG M L, WANG X X, LI S, et al. Joint offloading decision and resource allocation in mobile edge computing-enabled satellite-terrestrial network[J]. Symmetry, 2022, 14(3): 564.

XIN K J, FEI X, LIN P N. Research on task offloading strategy based on stackelberg game in LEO satellite network[J]. Journal of Xi'an Technological University, 2023, 43(4): 393-404.

WANG D Z, WANG W, KANG Y H, et al. Distributed data offloading in ultra-dense LEO satellite networks: a stackelberg mean-field game approach[J]. IEEE Journal of Selected Topics in Signal Processing, 2023, 17(1): 112-127.

XU C, ZHENG G Y, TANG L R. Energy-aware user association for NOMA-based mobile edge computing using matching-coalition game[J]. IEEE Access, 2020, 8: 61943-61955.

夏玮玮, 胡静, 宋铁成. 低地球轨道卫星边缘计算场景中任务卸载与资源分配联合优化算法[J]. 通信学报, 2024, 45(07): 48-60.

XIA W W, HU J, SONG T C. Joint optimization algorithm for task offloading and resource allocation in low earth orbit satellites edge computing scenario[J]. Journal of Communication, 2024, 45(07): 48-60.

HE Q, CUI G M, ZHANG X Y, et al. A game-theoretical approach for user allocation in edge computing environment[J]. IEEE Transactions on Parallel and Distributed Systems, 2020, 31(3): 515-529.

HU S H, LI G H. Dynamic request scheduling optimization in mobile edge computing for IoT applications[J]. IEEE Internet of Things Journal, 2020, 7(2): 1426-1437.

GAO X Q, LIU R K, KAUSHIK A. Virtual network function placement in satellite edge computing with a potential game approach[J]. IEEE Transactions on Network and Service Management, 2022, 19(2): 1243-1259.

IM G, JUNG D H, RYU J G. Enhancing the Connectivity of Satellite IoT Devices in Terrestrial-Non terrestrial Integrated Networks based on the Stackelberg Game Approach[C]//Proceedings of the 2020 International Conference on Information and Communication Technology Convergence (ICTC). Piscataway: IEEE Press, 2020: 1783-1785.

FENG R, LIN Z Y, WU P, et al. Potential game based distributed task scheduling for multi-earth observation satellites[C]//Proceedings of the 2020 7th International Conference on Information, Cybernetics, and Computational Social Systems (ICCSS). Piscataway: IEEE Press, 2020: 411-416.

金堃, 邓向阳, 于柯远. 深度强化学习之近端策略优化研究[J]. 物联网技术, 2023, 13(7): 69-75.

JIN K, DENG X Y, YU K Y. Research on near-end strategy optimization of deep reinforcement learning[J]. Internet of Things Technologies, 2023, 13(7): 69-75.

MALEKI H, BAŞARAN M, DURAK-ATA L. Handover-enabled dynamic computation offloading for vehicular edge computing networks[J]. IEEE Transactions on Vehicular Technology, 2023, 72(7): 9394-9405.

CAO Z L, ZHOU P, LI R X, et al. Multiagent deep reinforcement learning for joint multichannel access and task offloading of mobile-edge computing in industry 4.0[J]. IEEE Internet of Things Journal, 2020, 7(7): 6201-6213.

MUSTAFA E, SHUJA J, BILAL K, et al. Reinforcement learning for intelligent online computation offloading in wireless powered edge networks[J]. Cluster Computing, 2023, 26(2): 1053-1062.

YANG Y J, CHEN X, CHEN Y, et al. Green-oriented offloading and resource allocation by reinforcement learning in MEC[C]//Proceedings of the 2019 IEEE International Conference on Smart Internet of Things (SmartIoT). Piscataway: IEEE Press, 2019: 378-382.

MNIH V, KAVUKCUOGLU K, SILVER D, et al. Playing atari with deep reinforcement learning[EB ] . arXiv preprint, 2013, arXiv: 1312.5602 .

LIU J H, ZHAO B K, XIN Q, et al. DRL-ER: an intelligent energy-aware routing protocol with guaranteed delay bounds in satellite mega-constellations[J]. IEEE Transactions on Network Science and Engineering, 2021, 8(4): 2872-2884.

CHEN S W, YAO Z, JIANG X F, et al. Multi-agent deep reinforcement learning-based cooperative edge caching for ultra-dense next-generation networks[J]. IEEE Transactions on Communications, 2021, 69(4): 2441-2456.

WANG S Y, HUANG Y, CLERCKX B. Dynamic air-ground collaboration for multi-access edge computing[C]//Proceedings of the ICC 2022 - IEEE International Conference on Communications. Piscataway: IEEE Press, 2022: 5365-5371.

LI Y, LUO J T, RAN Y Y, et al. DeepISL: joint optimization of LEO inter-satellite link planning and power allocation via parameterized deep reinforcement learning[C]//Proceedings of the 2023 IEEE Global Communications Conference. Piscataway: IEEE Press, 2023: 3977-3982.

WANG C, WANG H W, WANG W D. A two-hops state-aware routing strategy based on deep reinforcement learning for LEO satellite networks[J]. Electronics, 2019, 8(9): 920.

刘治国, 张姣姣, 潘成胜. 基于Dueling-DDQN的星上带宽资源预留算法研究[J]. 兵器装备工程学报, 2023, 44(12): 272-277, 284.

LIU Z G, ZHANG J J, PAN C S. Research on on-board bandwidth resource reservation Algorithm based on Dueling DDQN[J]. Journal of Ordnance Engineering, 2023, 44(12): 272-277, 284.

BIRABWA D J, RAMOTSOELA D, VENTURA N. Multi-agent deep reinforcement learning for user association and resource allocation in integrated terrestrial and non-terrestrial networks[J]. Computer Networks, 2023, 231: 109827.

HSU C K. A dueling DQN-based computational offloading method in MEC-enabled IIoT network[J]. The Computer Journal, 2023, 66(12): 2887-2896.

CHEN N, ZHANG S, QIAN Z Z, et al. When learning joins edge: real-time proportional computation offloading via deep reinforcement learning[C]//Proceedings of the 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS). Piscataway: IEEE Press, 2019: 414-421.

LI J, GAO H, LV T J, et al. Deep reinforcement learning based computation offloading and resource allocation for MEC[C]//Proceedings of the 2018 IEEE Wireless Communications and Networking Conference (WCNC). Piscataway: IEEE Press, 2018: 1-6.

左珮良, 侯少龙, 郭超, 等. 基于强化学习的多层卫星网络边缘安全决策方法[J]. 通信学报, 2022, 43(6): 189-199.

ZUO P L, HOU S L, GUO C, et al. Security decision method for the edge of multi-layer satellite network based on reinforcement learning[J]. Journal on Communications, 2022, 43(6): 189-199.

CUI G F, LI X Y, XU L X, et al. Latency and energy optimization for MEC enhanced SAT-IoT networks[J]. IEEE Access, 2020, 8: 55915-55926.

PANG S C, ZHENG J Y, WANG M, et al. Minimize average tasks processing time in satellite mobile edge computing systems via a deep reinforcement learning method[J]. Journal of Cloud Computing, 2023, 12(1): 159.

李天宇. 基于强化学习的云计算资源调度策略研究[J]. 上海电力学院学报, 2019, 35(4): 399-403.

LI T Y. Research on cloud computing resource scheduling strategy based on reinforcement learning[J]. Journal of Shanghai University of Electric Power, 2019, 35(4): 399-403.

JIANG K, ZHOU H, LI D W, et al. A Q-learning based method for energy-efficient computation offloading in mobile edge computing[C]//Proceedings of the 2020 29th International Conference on Computer Communications and Networks (ICCCN). Piscataway: IEEE Press, 2020: 1-7.

彭明阳. 面向低轨星座的跳波束机制及资源分配研究[D]. 南京: 南京邮电大学, 2023.

PENG M Y. Research on beam hopping mechanism and resource allocation for low orbit constellation[D]. Nanjing: Nanjing University of Posts and Telecommunications, 2023.

张沛, 刘帅军, 马治国, 等. 基于深度增强学习和多目标优化改进的卫星资源分配算法[J]. 通信学报, 2020, 41(6): 51-60.

ZHANG P, LIU S J, MA Z G, et al. Improved satellite resource allocation algorithm based on DRL and MOP[J]. Journal on Communications, 2020, 41(6): 51-60.

ZHANG Y C, HU X, CHEN R, et al. Dynamic beam hopping for DVB-S2X satellite: a multi-objective deep reinforcement learning approach[C]//Proceedings of the 2019 IEEE International Conferences on Ubiquitous Computing & Communications (IUCC) and Data Science and Computational Intelligence(DSCI) and Smart Computing, Networking and Services(SmartCNS). Piscataway: IEEE Press, 2019: 164-169.

ZHANG P Y, CUI Z X, KUMAR N, et al. Local search resource allocation algorithm for space-based backbone network in Deep Reinforcement Learning method[J]. Ad Hoc Networks, 2024, 163: 103575.

YAO Y, DIAO X B, WANG X D, et al. Joint offloading strategy and trajectory optimization for aerial-ground cooperative mobile edge computing systems[C]//Proceedings of the IET 8th International Conference on Wireless, Mobile & Multimedia Networks. London: IET, 2019: 31-37.

NIE Y W, ZHAO J H, GAO F F, et al. Semi-distributed resource management in UAV-aided MEC systems: a multi-agent federated reinforcement learning approach[J]. IEEE Transactions on Vehicular Technology, 2021, 70(12): 13162-13173.

LAI J Y, LIU H S, SUN Y S, et al. Multi-agent deep reinforcement learning aided computing offloading in LEO satellite networks[C]//Proceedings of the ICC 2023 - IEEE International Conference on Communications. Piscataway: IEEE Press, 2023: 3438-3443.

WANG X T, DAI Z Q, XU Z. LEO satellite network routing algorithm based on reinforcement learning[C]//Proceedings of the 2021 IEEE 4th International Conference on Electronics Technology (ICET). Piscataway: IEEE Press, 2021: 1105-1109.

HSU Y H, LEE J N, XU F M. A deep reinforcement learning based routing scheme for LEO satellite networks in 6G[C]//Proceedings of the 2023 IEEE Wireless Communications and Networking Conference (WCNC). Piscataway: IEEE Press, 2023: 1-6.

HUANG Y X, WU S F, KANG Z Y, et al. Reinforcement learning based dynamic distributed routing scheme for mega LEO satellite networks[J]. Chinese Journal of Aeronautics, 2023, 36(2): 284-291.

MA T, XIN X R, MA X, et al. An intelligent multiple access protocol: federated transfer reinforcement learning for satellite-ground network[C]//Proceedings of the 2023 IEEE 11th International Conference on Information, Communication and Networks (ICICN). Piscataway: IEEE Press, 2023: 218-223.

NATH S, WU J X. Dynamic computation offloading and resource allocation for multi-user mobile edge computing[C]//Proceedings of the 2020 IEEE Global Communications Conference. Piscataway: IEEE Press, 2020: 1-6.

CAO S H, CHEN S, CHEN H, et al. HCOME: research on hybrid computation offloading strategy for MEC based on DDPG[J]. Electronics, 2023, 12(3): 562.

YAN J, BI S Z, ZHANG Y J A. Offloading and resource allocation with general task graph in mobile edge computing: a deep reinforcement learning approach[J]. IEEE Transactions on Wireless Communications, 2020, 19(8): 5404-5419.

LI H, YU J Y, CAO L L, et al. Multi-agent reinforcement learning based computation offloading and resource allocation for LEO Satellite edge computing networks[J]. Computer Communications, 2024, 222: 268-276.

谭惠东. 低轨星座宽带网络边缘计算任务调度策略研究[D]. 成都: 电子科技大学, 2023.

TAN H D. Research on edge computing task scheduling strategy for low-orbit constellation broadband networks[D]. Chengdu: University of Electronic Science and Technology of China, 2023.

周碧莹. 基于强化学习的中低轨卫星网络系统资源调度技术研究[D]. 合肥: 安徽大学, 2020.

ZHOU B Y. Research on Resource scheduling technology of low-medium orbit satellite network system based on reinforcement learning [D]. Anhui University, 2020, Anhui, China.

LIN Z Y, NI Z Y, KUANG L L, et al. Dynamic beam pattern and bandwidth allocation based on multi-agent deep reinforcement learning for beam hopping satellite systems[J]. IEEE Transactions on Vehicular Technology, 2022, 71(4): 3917-3930.

ZHU J C, SUN X B, CHEN D J. Deep reinforcement learning for congestion control and routing optimization in LEO satellite networks[C]//Proceedings of the 2023 5th International Conference on Artificial Intelligence and Computer Applications (ICAICA). Piscataway: IEEE Press, 2023: 739-742.

LEEL J H, SELVAM D P, MOLISCH A F, et al. Reinforcement learning empowered massive IoT access in LEO-based non-terrestrial networks[C]//Proceedings of the 2022 13th International Conference on Information and Communication Technology Convergence (ICTC). Piscataway: IEEE Press, 2022: 1347-1350.

韩明仁, 王玉峰. 基于强化学习的全电推进卫星变轨优化方法[J]. 系统工程与电子技术, 2022, 44(5): 1652-1661.

HAN M R, WANG Y F. Optimization method for orbit transfer of all-electric propulsion satellite based on reinforcement learning[J]. Systems Engineering and Electronics, 2022, 44(5): 1652-1661.

XU J, FAN S M, ZHAO Z T, et al. Deep reinforcement learning-based dynamic multi-beam power allocation for GEO-LEO co-existing satellites[J]. Acta Astronautica, 2024, 223: 197-209.

秦泽宇. 低轨卫星物联网中任务卸载策略优化方法研究[D]. 北京: 北京邮电大学, 2022.

QIN Z Y. Research on Optimization Method of mission unloading Strategy in Low-orbit Satellite Internet of Things[D]. Beijing: Beijing University of Posts and Telecommunications, 2022.

WANG Q T, CHEN S Y, ZONG J Y, et al. Resource Allocation in LEO Satellites assisted Terrestrial Network for Latency-Sensitive Task[C]//Proceedings of the 2024 International Wireless Communications and Mobile Computing(IWCMC). Piscataway: IEEE Press, 2024: 836-841.

ZHU Y, LU D W. Satellite-assisted edge computing management based on deep reinforcement learning in industrial Internet of Things[J]. Computer Networks, 2023, 237: 110050.

HUANG J F, YANG Y, YIN L F, et al. Deep reinforcement learning-based power allocation for rate-splitting multiple access in 6G LEO satellite communication system[J]. IEEE Wireless Communications Letters, 2022, 11(10): 2185-2189.

OU J W, XING L N, YAO F, et al. Deep reinforcement learning method for satellite range scheduling problem[J]. Swarm and Evolutionary Computation, 2023, 77: 101233.

DURYEA E, GANGER M, HU W. Exploring deep reinforcement learning with multi Q-learning[J]. Intelligent Control and Automation, 2016, 7(4): 129-144.

LIN X, CHEN Y N, XUE J H, et al. Large-volume LEO satellite imaging data networked transmission scheduling problem: model and algorithm[J]. Expert Systems with Applications, 2024, 249: 123649.

LI X H,CHEN B,KANG J W, et al. Coalition game-assisted joint resource optimization for digital twin-assisted edge intelligence[J]. Computer Science, 2023, 50(2): 42-49.

吴树范, 王伟, 温济帆, 等. 低轨互联网星座发展研究综述[J]. 北京航空航天大学学报, 2024, 50(1): 1-11.

WU S F, WANG W, WEN J F, et al. Review on the development of low-orbit Internet constellation[J]. Journal of Beijing University, 2024, 50(1): 1-11.

王雨晗. 基于卫星边缘计算的任务卸载与服务放置策略研究[D]. 北京: 北京邮电大学, 2023.

WANG Y H. Research on mission unloading and service placement Strategy based on satellite edge computing[D]. Beijing: Beijing University of Posts and Telecommunications, 2023.

WEI P, FENG W, WANG Y M, et al. Joint mobility control and MEC offloading for hybrid satellite-terrestrial-network-enabled robots[J]. IEEE Transactions on Wireless Communications, 2023, 22(11): 8483-8497.

ZHU X M, JIANG C X. Delay optimization for cooperative multi-tier computing in integrated satellite-terrestrial networks[J]. IEEE Journal on Selected Areas in Communications, 2023, 41(2): 366-380.

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

面向化工园区有害气体泄漏的博弈巡逻策略

面向KubeEdge边缘计算系统应用研究

基于边缘计算的云原生机器人系统

复杂微电网控制中的随机博弈与优化研究

Cerberus：基于深度学习的跨站社交网络机器人检测系统