深度强化学习应用于金融市场量化交易研究综述

许波; 贺一峻; 温健城; 李祥霞

doi:10.11959/j.issn.2096-6652.202439

您当前的位置：

首页 >

文章列表页 >

深度强化学习应用于金融市场量化交易研究综述

综述与展望 | 更新时间：2025-01-24

- 深度强化学习应用于金融市场量化交易研究综述
- Review: the application of deep reinforcement learning to quantitative trading in financial market
- 智能科学与技术学报 2024年6卷第4期页码：416-428
- 作者机构：
  
  1.广东财经大学信息学院，广东广州 510320
  2.广东省智能商务工程技术研究中心，广东广州 510320
- 作者简介：
  
  [ "许波（1982- ），男，博士，广东财经大学信息学院副教授，CCF高级会员，计算机应用专委会执行委员，主要研究方向为机器学习、多智能体等。" ]
  [ "贺一峻（1998- ），男，广东财经大学信息学院硕士生，CCF学生会员，主要研究方向为机器学习、多智能体。" ]
  [ "温健城（2001- ），男，广东财经大学信息学院硕士生，CCF学生会员，主要研究方向为机器学习、多智能体。" ]
  [ "李祥霞（1988- ），女，博士，广东财经大学信息学院副教授，主要研究方向为机器学习、深度学习等。" ]
- 基金信息：
  
  全国统计科学研究项目(2024LY075);教育部人文社科项目(24YJAZH187);广东省哲学社会科学规划项目(GD24CGL08)
- DOI：10.11959/j.issn.2096-6652.202439
  中图分类号： TP39
- 收稿：2024-08-07，
  
  修回：2024-10-11，
  
  纸质出版：2024-12-15
- 稿件说明：
移动端阅览
许波,贺一峻,温健城等.深度强化学习应用于金融市场量化交易研究综述[J].智能科学与技术学报,2024,06(04):416-428.

XU Bo,HE Yijun,WEN Jiancheng,et al.Review: the application of deep reinforcement learning to quantitative trading in financial market[J].Chinese Journal of Intelligent Science and Technology,2024,06(04):416-428.
许波,贺一峻,温健城等.深度强化学习应用于金融市场量化交易研究综述[J].智能科学与技术学报,2024,06(04):416-428. DOI： 10.11959/j.issn.2096-6652.202439.

XU Bo,HE Yijun,WEN Jiancheng,et al.Review: the application of deep reinforcement learning to quantitative trading in financial market[J].Chinese Journal of Intelligent Science and Technology,2024,06(04):416-428. DOI： 10.11959/j.issn.2096-6652.202439.

摘要

深度强化学习（deep reinforcement learning，DRL）作为一种实现通用人工智能的有效学习范式，已在一系列实际金融市场量化交易应用中取得了显著成果，成为该领域的主流方法。首先，对深度强化学习的基本概念和原理进行了详细介绍，在此基础上，系统地综述了DRL在金融市场量化交易中的应用和实践进展，涵盖了基于策略的算法模型、基于价值的算法模型，以及基于演员—评论家的算法模型等不同类型的DRL在金融市场量化交易中的应用。进一步探讨了DRL在金融市场量化交易中的优势，指出其能够根据市场环境的动态变化调整交易策略，以适应不同的市场环境。其次，指出了DRL在金融市场量化交易中面临的挑战，包括数据质量问题、模型稳定性问题、过拟合问题等。最后，对DRL在金融市场量化交易领域未来的发展趋势进行了展望。随着算法的不断优化和计算能力的提升，DRL将在金融市场量化交易领域发挥更加重要的作用，为投资决策提供更加准确和可靠的支持。

Abstract

As an effective learning paradigm to realize general artificial intelligence

deep reinforcement learning (DRL) has achieved significant results in a series of practical quantitative trading applications in financial market

becoming the mainstream method in this field. Firstly

a detailed introduction to the basic concepts and principles of deep reinforcement learning were provided. On this basis

a systematic review was conducted on the application and practical progress of DRL in quantitative trading in financial market

covering the application of different types of DRL

such as policy-based algorithm models

value-based algorithm models

and actor-critic algorithm models in quantitative trading in financial market. The advantages of DRL in quantitative trading in financial market were further explored

pointing out DRL could adjust trading strategies based on dynamic changes in the market environments to adapt to different market environments. Secondly

the challenges faced by DRL in quantitative trading in financial market were pointed out

including data quality issues

model stability issues

overfitting issues

etc. Finally

we outlooked the future development trend of DRL in the field of quantitative trading in financial market. It is believed that with the continuous optimization of algorithms and the improvement of computing power

DRL will play a more important role in the field of quantitative trading in financial market

providing more accurate and reliable support for investment decisions.

关键词

Keywords

references

FILAR J, VRIEZE K. Competitive markov decision processes[M]. New York: Springer, 1996.

刘朝阳, 穆朝絮, 孙长银. 深度强化学习算法与应用研究现状综述[J]. 智能科学与技术学报, 2020, 2(4): 314-326.

LIU Z Y, MU C X, SUN C Y. An overview on algorithms and applications of deep reinforcement learning[J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4): 314-326.

HESTER T, VECERIK M, PIETQUIN O, et al. Deep Q -learning from demonstrations[J ] . Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1).

GAŠPEROV B, ŠARIĆ F, BEGUŠIĆ S, et al. Adaptive rolling window selection for minimum variance portfolio estimation based on reinforcement learning[C]//Proceedings of the 2020 43rd International Convention on Information, Communication and Electronic Technology (MIPRO). Piscataway: IEEE Press, 2020: 1098-1102.

HAN L, DING N, WANG G, et al. Efficient continuous space policy optimization for high-frequency trading[C]//Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. New York: ACM, 2023: 4112-4122.

BORRAGEIRO G, FIROOZYE N, BARUCCA P. Reinforcement learning for systematic FX trading[J]. IEEE Access, 2022, 10: 5024-5036.

MATTERA G, MATTERA R. Shrinkage estimation with reinforcement learning of large variance matrices for portfolio selection[J]. Intelligent Systems with Applications, 2023, 17: 200181.

LIU F R, LI Y, LI B T, et al. Bitcoin transaction strategy construction based on deep reinforcement learning[J]. Applied Soft Computing, 2021, 113: 107952.

DING F Q, LUO C. An adaptive financial trading system using deep reinforcement learning with candlestick decomposing features[J]. IEEE Access, 2020, 8: 63666-63678.

赵淼, 谢良, 林文静, 等. 基于动态选择预测器的深度强化学习投资组合模型[J]. 计算机科学, 2023: 1-14.

ZHAO M, XIE L, LIN W J, et al. A deep reinforcement learning portfolio model based on dynamic selectors[J]. Computer Science, 2023: 1-14.

DUAN Z J, CHEN C, CHENG D W, et al. Optimal action space search: an effective deep reinforcement learning method for algorithmic trading[C]//Proceedings of the 31st ACM International Conference on Information & Knowledge Management. New York: ACM, 2022: 406-415.

LI J H, ZHANG Y, YANG X Y, et al. Online portfolio management via deep reinforcement learning with high-frequency data[J]. Information Processing & Management, 2023, 60(3): 103247.

CHUNG G, CHUNG M, LEE Y, et al. Market making under order stacking framework: a deep reinforcement learning approach[C]//Proceedings of the Third ACM International Conference on AI in Finance. New York: ACM, 2022: 223-231.

CHEN X M, GUO H R. A futures quantitative trading strategy based on a deep reinforcement learning algorithm[C]//Proceedings of the 2023 IEEE 8th International Conference on Big Data Analytics (ICBDA). Piscataway: IEEE Press, 2023: 175-179.

ALAMEER A, AL SHEHRI K. Conditional value-at-risk for quantitative trading: a direct reinforcement learning approach[C]//Proceedings of the 2022 IEEE Conference on Control Technology and Applications (CCTA). Piscataway: IEEE Press, 2022: 1208-1213.

CUI T X, DING S S, JIN H, et al. Portfolio constructions in cryptocurrency market: a CVaR-based deep reinforcement learning approach[J]. Economic Modelling, 2023, 119: 106078.

许杰, 祝玉坤, 邢春晓. 基于深度强化学习的金融交易算法研究[J]. 计算机工程与应用, 2022, 58(7): 276-285.

XU J, ZHU Y K, XING C X. Research on financial trading algorithm based on deep reinforcement learning[J]. Computer Engineering and Applications, 2022, 58(7): 276-285.

SULIMAN U, VAN ZYL T L, PASKARAMOORTHY A. Cryptocurrency trading agent using deep reinforcement learning[C]//Proceedings of the 2022 9th International Conference on Soft Computing & Machine Intelligence (ISCMI). Piscataway: IEEE Press, 2022: 6-10.

WANG J J, JING F, HE M L. Stock trading strategy of reinforcement learning driven by turning point classification[J]. Neural Processing Letters, 2023, 55(3): 3489-3508.

杨兴雨, 陈亮威, 郑萧腾, 等. 考虑行为克隆的深度强化学习股票交易策略[J]. 系统管理, 2023: 1-17.

YANG X Y, CHEN L W, ZHENG X T, et al. Stock trading strategy via deep reinforcement learning with behavior cloning[J]. Journal of Systems & Management, 2023: 1-17.

MA C, ZHANG J S, LIU J M, et al. A parallel multi-module deep reinforcement learning algorithm for stock trading[J]. Neurocomputing, 2021, 449: 290-302.

THÉATE T, ERNST D. An application of deep reinforcement learning to algorithmic trading[J]. Expert Systems with Applications, 2021, 173: 114632.

LIU P P, ZHANG Y F, BAO F X, et al. Multi-type data fusion framework based on deep reinforcement learning for algorithmic trading[J]. Applied Intelligence, 2023, 53(2): 1683-1706.

CANNELLI L, NUTI G, SALA M, et al. Hedging using reinforcement learning: Contextual k-armed bandit versus Q -Learning[J ] . The Journal of Finance and Data Science, 2023, 9: 100101.

HUANG Z, GONG W Y, DUAN J W. TBDQN: a novel two-branch deep Q-network for crude oil and natural gas futures trading[J]. Applied Energy, 2023, 347: 121321.

ZHAO P C, YIN S Y, KUMAR B V D, et al. Improving quantitative stock trading prediction based on MAD using Q-learning technology[C]//Proceedings of the 2023 International Conference on Artificial Intelligence and Smart Communication (AISC). Piscataway: IEEE Press, 2023: 57-60.

KONDA V, TSITSIKLIS J. Actor-critic algorithms[J ] . Advances in Neural Information Processing Systems, 1999, 12 .

XU F C, TAN S. Dynamic portfolio management based on pair trading and deep reinforcement learning[C]//Proceedings of the 2020 3rd International Conference on Computational Intelligence and Intelligent Systems. New York: ACM, 2020: 50-55.

GAO N, HE Y Y, JIAO Y M, et al. Online optimal investment portfolio model based on deep reinforcement learning[C]//Proceedings of the 2021 13th International Conference on Machine Learning and Computing. New York: ACM, 2021: 14-20.

LIM Q Y E, CAO Q, QUEK C. Dynamic portfolio rebalancing through reinforcement learning[J]. Neural Computing and Applications, 2022, 34(9): 7125-7139.

KORATAMADDI P, WADHWANI K, GUPTA M, et al. Market sentiment-aware deep reinforcement learning approach for stock portfolio allocation[J]. Engineering Science and Technology, an International Journal, 2021, 24(4): 848-859.

ZHAO T L, MA X, LI X M, et al. Asset correlation based deep reinforcement learning for the portfolio selection[J]. Expert Systems with Applications, 2023, 221: 119707.

WENG L G, SUN X D, XIA M, et al. Portfolio trading system of digital currencies: a deep reinforcement learning with multidimensional attention gating mechanism[J]. Neurocomputing, 2020, 402: 171-182.

MURRAY P, WOOD B, BUEHLER H, et al. Deep hedging: continuous reinforcement learning for hedging of general portfolios across multiple risk aversions[C]//Proceedings of the Third ACM International Conference on AI in Finance. New York: ACM, 2022: 361-368.

WANG Z Y, JIN S R, LI W. Research on portfolio optimization based on deep reinforcement learning[C]//Proceedings of the 2022 4th International Conference on Machine Learning, Big Data and Business Intelligence (MLBDBI). Piscataway: IEEE Press, 2022: 391-395.

HUANG S H, MIAO Y H, HSIAO Y T. Novel deep reinforcement algorithm with adaptive sampling strategy for continuous portfolio optimization[J]. IEEE Access, 2021, 9: 77371-77385.

JANG J, SEONG N. Deep reinforcement learning for stock portfolio optimization by connecting with modern portfolio theory[J]. Expert Systems with Applications, 2023, 218: 119556.

冀中, 张文嘉. 集成深度强化学习在股票指数投资组合优化中的应用分析[J]. 计算机科学与探索, 2024: 1-10.

JI Z, ZHANG W J. Application analysis of integrated deep reinforcement learning in stock index portfolio optimization[J]. Computer Science and Exploration, 2024: 1-10.

YUAN X K, JIANG H Q, HUANG L Y. DRL-based quantitative algorithms for gold and Bitcoin portfolio decisions[C]//Proceedings of the 2022 IEEE International Conference on Advances in Electrical Engineering and Computer Applications (AEECA). Piscataway: IEEE Press, 2022: 878-881.

LUTHFIANTI B A, SAEPUDIN D, IHSAN A F. Portfolio allocation of stocks in index LQ45 using deep reinforcement learning[C]//Proceedings of the 2022 10th International Conference on Information and Communication Technology (ICoICT). Piscataway: IEEE Press, 2022: 205-210.

SUN Q Z, SI Y W. Supervised actor-critic reinforcement learning with action feedback for algorithmic trading[J]. Applied Intelligence, 2023, 53(13): 16875-16892.

ZHU Z Z. Research on quantitative timing trading strategy based on deep reinforcement learning[C]//Proceedings of the 2022 2nd International Conference on Consumer Electronics and Computer Engineering (ICCECE). Piscataway: IEEE Press, 2022: 633-638.

CHEN J C, CHEN C X, DUAN L J, et al. DDPG based on multi-scale strokes for financial time series trading strategy[C]//Proceedings of the 2022 8th International Conference on Computer Technology Applications. New York: ACM, 2022: 22-27.

LIN Y C, CHEN C T, SANG C Y, et al. Multiagent-based deep reinforcement learning for risk-shifting portfolio management[J]. Applied Soft Computing, 2022, 123: 108894.

MA C, ZHANG J S, LI Z X, et al. Multi-agent deep reinforcement learning algorithm with trend consistency regularization for portfolio management[J]. Neural Computing and Applications, 2023, 35(9): 6589-6601.

龙杰, 谢良, 徐海蛟. 集成的深度强化学习投资组合模型[J]. 计算机应用, 2024, 44(1): 300-310.

LONG J, XIE L, XU H J. Integrated deep reinforcement learning portfolio model[J]. Journal of Computer Applications, 2024, 44(1): 300-310.

BELYAKOV B, SIZYKH D. Deep reinforcement learning task for portfolio construction[C]//Proceedings of the 2021 International Conference on Data Mining Workshops (ICDMW). Piscataway: IEEE Press, 2021: 1077-1082.

ZHANG Y F, ZHAO P L, WU Q Y, et al. Cost-sensitive portfolio selection via deep reinforcement learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 34(1): 236-248.

ZHANG W P, ZHANG N, YAN J C, et al. Auto uning of price prediction models for high-frequency trading via reinforcement learning[J]. Pattern Recognition, 2022, 125: 108543.

WU X, CHEN H L, WANG J J, et al. Adaptive stock trading strategies with deep reinforcement learning methods[J]. Information Sciences, 2020, 538: 142-158.

OLSCHEWSKI S, DIAO L N, RIESKAMP J. Reinforcement learning about asset variability and correlation in repeated portfolio decisions[J]. Journal of Behavioral and Experimental Finance, 2021, 32: 100559.

NICOLAS P G, PAUL-ANTOINE B. Deep hierarchical reinforcement learning to manage the trade-off between sustainability and profitability in common pool resources systems[C]//Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN). Piscataway: IEEE Press, 2021: 1-7.

GAŠPEROV B, KOSTANJČAR Z. Market making with signals through deep reinforcement learning[J]. IEEE Access, 2021, 9: 61611-61622.

SONG Z T, WANG Y N, QIAN P, et al. From deterministic to stochastic: an interpretable stochastic model-free reinforcement learning framework for portfolio optimization[J]. Applied Intelligence, 2023, 53(12): 15188-15203.

YE Y N, PEI H Z, WANG B X, et al. Reinforcement-learning based portfolio management with augmented asset movement prediction states[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(1): 1112-1119.

LI X J, CUI C R, CAO D L, et al. Hypergraph-based reinforcement learning for stock portfolio selection[C]//Proceedings of the ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Piscataway: IEEE Press, 2022: 4028-4032.

卜湛, 张善凡, 李雪延, 等. 基于深度强化学习的自适应股指预测研究[J]. 管理科学学报, 2023, 26(4): 148-174.

BU Z, ZHANG S F, LI X Y, et al. Adaptive stock index prediction based on deep reinforcement learning[J]. Journal of Management Sciences in China, 2023, 26(4): 148-174.

WANG Z C, HUANG B W, TU S K, et al. DeepTrader: a deep reinforcement learning approach for risk-return balanced portfolio management with market conditions embedding[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(1): 643-650.

SOLEYMANI F, PAQUET E. Deep graph convolutional reinforcement learning for financial portfolio management-DeepPocket[J]. Expert Systems with Applications, 2021, 182: 115127.

SHI S, LI J J, LI G H, et al. GPM: a graph convolutional network based reinforcement learning framework for portfolio management[J]. Neurocomputing, 2022, 498: 14-27.

YANG M, ZHENG X, LIANG Q, et al. A smart trader for portfolio management based on normalizing flows[C]//Proceedings of the IJCAI. [S.l.: s.n.], 2022: 4014-4021.

NIU H, LI S Y, LI J. MetaTrader: an reinforcement learning approach integrating diverse policies for portfolio optimization[C]//Proceedings of the 31st ACM International Conference on Information & Knowledge Management. New York: ACM, 2022: 1573-1583.

柴洪峰, 王帅, 涂晓军, 等. 智能化金融科技创新监管工具: 理念、平台框架和展望[J]. 智能科学与技术学报, 2020, 2(3): 214-226.

CHAI H F, WANG S, TU X J, et al. Intelligent innovative regulatory tools on financial technology: concept, platform framework, and prospects[J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(3): 214-226.

浏览量

1175

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

面向数字货币量化交易的OAC模型研究

基于KPPO算法的四旋翼无人机飞行控制

人工智能与量化交易的课程建设

基于双池DQN的HVAC无模型优化控制方法

基于TD3的电动汽车复合电源能量管理策略研究