- 个人中心
- 退出登录
浏览全部资源
扫码关注微信
A DQN-based approach for energy-efficient train driving control
- Chinese Journal of Intelligent Science and Technology Vol. 2, Issue 4, Pages: 372-384(2020)
- 作者机构：
1. 北京交通大学轨道交通控制与安全国家重点实验室，北京 100044
2. 中国科学院自动化研究所复杂系统管理与控制国家重点实验室，北京 100190
- 作者简介：
- 基金信息：
DOI：10.11959/j.issn.2096-6652.202040
CLC： U238.4-
Published Online：2020-12，
Published：15 December 2020
- 稿件说明：
移动端阅览
Shuai SU, Qingyang ZHU, Qinglai WEI, et al. A DQN-based approach for energy-efficient train driving control[J]. Chinese journal of intelligent science and technology, 2020, 2(4): 372-384.
Shuai SU, Qingyang ZHU, Qinglai WEI, et al. A DQN-based approach for energy-efficient train driving control[J]. Chinese journal of intelligent science and technology, 2020, 2(4): 372-384. DOI： 10.11959/j.issn.2096-6652.202040.
摘要
随着轨道交通网络规模的扩大和列车运营间隔的缩短，列车牵引能耗在快速增加。因此，通过优化列车的驾驶策略降低牵引能耗，对于轨道交通系统的节能减排具有重大意义。针对列车的驾驶策略优化问题，提出一种基于深度 Q 网络（DQN）的列车节能驾驶控制方法。首先介绍了传统的列车节能驾驶问题并构造其反问题，即通过分配最少的能耗达到规定运行时分。进一步将该问题转化为有限马尔可夫决策过程（MDP），通过设计状态动作值函数、定义动作策略选取方法等，构建基于 DQN 方法的列车节能驾驶控制方法。通过实际驾驶数据对DQN 进行训练，得到最优的状态动作值函数，并通过该值函数确定最优的能耗分配方案，从而得到最优驾驶策略。最后，以北京地铁亦庄线的实际运营数据设计了仿真算例，对方法的有效性进行验证，并对方法参数进行了敏感度分析。提出的方法可充分利用列车的驾驶数据提升驾驶策略，降低列车牵引能耗，对未来我国智慧城轨的发展具有一定的借鉴意义。
Abstract
The energy consumption in railway system is growing rapidly due to the expanding scale of the railway network and decreased operational headway.Hence
it is of great significant to apply the energy-efficient operation of the vehicles to cut down the energy cost of the railway system.A method for solving the energy-efficient train driving control based on deep Q-network (DQN) approach was proposed.Firstly
the traditional energy-efficient train driving control problem was presented and its inverse problem was formulated
i.e.
distributing the least energy consumption units to achieve the scheduled trip time.Moreover
the problem was reformulated as a Markov decision process (MDP) and a DQN-based approach for energy-efficient train driving control was proposed.A DQN was built to approximate the action value function which determines the optimal energy distribution policy and further obtain the optimal driving strategy.Finally
a numerical experiment based on the real-world operational data was proposed to verify the effectiveness of the proposed method and analyze the performance of the proposed method.The driving data of the trains is applied to improve the driving strategy via the proposed method in the paper which reduces the traction energy consumption.It is of significance for the future development of Chinese intelligent urban railway system.
关键词
Keywords
references
中国城市轨道交通协会 . 2019年度中国内地城轨交通线路概况 [R ] . 2020 .
China Association of Metro . An overview of urban rail transit lines in the mainland of China in 2019 [R ] . 2020 .
褚心童 , 张亚东 , 郭进 , 等 . 基于蚁群算法的列车节能驾驶策略优化算法研究 [J ] . 铁道标准设计 , 2020 , 65 ( 2 ): 1 - 7 .
CHU X T , ZHANG Y D , GUO J , et al . An optimization algorithm for train energy-saving driving strategy based on ant colony algorithm [J ] . Railway Standard Design , 2020 , 65 ( 2 ): 1 - 7 .
YANG X , LI X , NING B , et al . A survey on energy-efficient train operation for urban rail transit [J ] . IEEE Transactions on Intelligent Transportation Systems , 2015 , 17 ( 1 ): 2 - 13 .
张钹 . 人工智能进入后深度学习时代 [J ] . 智能科学与技术学报 , 2019 , 1 ( 1 ): 4 - 6 .
ZHANG B . Artificial intelligence is entering the post deep-learning era [J ] . Chinese Journal of Intelligent Science and Technology , 2019 , 1 ( 1 ): 4 - 6 .
袁小锋 , 王雅琳 , 阳春华 , 等 . 深度学习在流程工业过程数据建模中的应用 [J ] . 智能科学与技术学报 , 2020 , 2 ( 2 ): 107 - 115 .
YUAN X F , WANG Y L , YANG C H , et al . The application of deep learning in data-driven modeling of process industries [J ] . Chinese Journal of Intelligent Science and Technology , 2020 , 2 ( 2 ): 107 - 115 .
ICHIKAWA K . Application of optimization theory for bounded state variable problems to the operation of train [J ] . Bulletin of JSME , 1968 , 11 ( 47 ): 857 - 865 .
HOWLETT P . Optimal strategies for the control of a train [J ] . Automatica , 1996 , 32 ( 4 ): 519 - 532 .
HOWLETT P . An optimal strategy for the control of a train [J ] . The Anziam Journal , 1990 , 31 ( 4 ): 454 - 471 .
LIU R R , GOLOVITCHER I M . Energy-efficient operation of rail vehicles [J ] . Transportation Research Part A:Policy and Practice , 2003 , 37 ( 10 ): 917 - 932 .
KHMELNITSKY E . On an optimal control problem of train operation [J ] . IEEE Transactions on Automatic Control , 2000 , 45 ( 7 ): 1257 - 1266 .
曲健伟 , 王青元 , 孙鹏飞 . 基于极大值原理的地铁列车节能驾驶简化算法 [J ] . 铁道科学与工程学报 , 2019 , 16 ( 6 ): 1577 - 1586 .
QU J W , WANG Q Y , SUN P F . A simplified algorithm of energy-efficient driving for metro trains based on the maximum principle [J ] . Journal of Railway Science and Engineering , 2019 , 16 ( 6 ): 1577 - 1586 .
ARADI S , BECSI T , GASPAR P . A predictive optimization method for energy-optimal speed profile generation for train [C ] // 14th IEEE International Symposium on Computational Intelligence and Informatics . Piscataway:IEEE Press , 2013 : 135 - 139 .
WANG Y H , SCHUTTER D , NING B , et al . Optimal trajectory planning for trains - a pseudo spectral method and a mixed integer linear programming approach [J ] . Transportation Research Part C:Emerging Technologies , 2013 , 29 : 97 - 114 .
SCHEEPMAKER G M , GOVERDE R M P . Energy-efficient train control including regenerative braking with catenary efficiency [C ] // 2016 IEEE International Conference on Intelligent Rail Transportation . Piscataway:IEEE Press , 2016 : 116 - 122 .
ALBRECHT A , HOWLETT P , PUDNEY P , et al . The key principles of optimal train control—part 1:formulation of the model,strategies of optimal type,evolutionary lines,location of optimal switching points [J ] . Transportation Research Part B:Methodological , 2016 , 94 : 482 - 508 .
ALBRECHT A , HOWLETT P , PUDNEY P , et al . The key principles of optimal train control—part 2:existence of an optimal strategy,the local energy minimization principle,uniqueness,computational techniques [J ] . Transportation Research Part B:Methodological , 2016 , 94 : 509 - 538 .
MORRIS B , FEDERICA F , MICHELA L . Application of genetic algorithms for driverless subway train energy optimization [J ] . International Journal of Vehicular Technology , 2016 : 1 - 14 .
DOMNGUEZ M , FERNÁNDEZ A , CUCALA A P , et al . Optimal design of metro automatic train operation speed profiles for reducing energy consumption [J ] . Proceedings of the Institution of Mechanical Engineers,Part F:Journal of Rail and Rapid Transit , 2011 , 225 ( 5 ): 463 - 474 .
DOMNGUEZ M , FERNÁNDEZ A , CUCALA A P , et al . Energy savings in metropolitan railway substations through regenerative energy recovery and optimal design of ATO speed profiles [J ] . IEEE Transactions on Automation Science and Engineering , 2012 , 9 ( 3 ): 496 - 504 .
ZHANG C Y , CHEN D W , YIN J T , et al . Data-driven train operation models based on data mining and driving experience for the diesel-electric locomotive [J ] . Advanced Engineering Informatics , 2016 , 30 ( 3 ): 553 - 563 .
LIU X , DAI S H . Optimization of train control strategy for energy saving and time precision using multi-objective cuckoo search algorithm [C ] // The 2nd International Conference on Computer Science and Application Engineering .[S.l.:s.n. ] , 2018 : 1 - 5 .
CHENG R J , CHEN D W , GAI W L , et al . Intelligent driving methods based on sparse LSSVM and ensemble CART algorithms for high-speed trains [J ] . Computers ＆ Industrial Engineering , 2019 , 127 : 1203 - 1213 .
厉高 , 林建辉 , 庄哲 , 等 . 基于再生制动的城市轨道列车节能控制研究 [J ] . 铁道运输与经济 , 2019 , 41 ( 3 ): 121 - 126 .
LI G , LIN J H , ZHUANG Z , et al . A research on energy-saving train control of urban mass transit based on regenerative brake [J ] . Railway Transport and Economy , 2019 , 41 ( 3 ): 121 - 126 .
HUANG Y N , TAN L T , CHEN L , et al . A neural network driving curve generation method for the heavy-haul train [J ] . Advances in Mechanical Engineering , 2016 , 8 ( 5 ): 1 - 14 .
YIN J T , CHEN D W , LI L X . Intelligent train operation algorithms for subway by expert system and reinforcement learning [J ] . IEEE Transactions on Intelligent Transportation Systems , 2014 , 15 ( 6 ): 2561 - 2571 .
ZHOU R , SONG S J . Optimal automatic train operation via deep reinforcement learning [C ] // The 10th International Conference on Advanced Computational Intelligence .[S.l.:s.n. ] , 2018 : 103 - 108 .
HUANG K , WU J J , YANG X , et al . Discrete train speed profile optimization for urban rail transit:a data-driven model and integrated algorithms based on machine learning [J ] . Journal of Advanced Transportation , 2019 , 4 : 1 - 17 .
HUANG J , ZHANG E D , ZHANG J R , et al . Deep reinforcement learning based train driving optimization [C ] // 2019 Chinese Automation Congress .[S.l.:s.n. ] 2019 .
ZHOU K C , SONG S J , XUE A K , et al . Smart train operation algorithms based on expert knowledge and reinforcement learning [J ] . IEEE Transactions on Systems,Man,and Cybernetics:Systems , 2020 : 1 - 12 .
QI X W , LUO Y D , WU G Y , et al . Deep reinforcement learning enabled self-learning control for energy efficient driving [J ] . Transportation Research Part C:Emerging Technologies , 2019 , 99 : 67 - 81 .
张淼 , 张琦 , 张梓轩 . 基于Q学习算法的高速铁路列车节能优化研究 [J ] . 铁道运输与经济 , 2019 , 41 ( 12 ): 111 - 117 .
ZHANG M , ZHANG Q , ZHANG Z X . A study on energy-saving optimization for high-speed railways train based on Q-learning algorithm [J ] . Railway Transport and Economy , 2019 , 41 ( 12 ): 111 - 117 .
SU S , TANG T , LI X . Driving strategy optimization for trains in subway systems [J ] . Proceedings of the Institution of Mechanical Engineers,Part F:Journal of Rail and Rapid Transit , 2018 , 232 ( 2 ): 369 - 383 .
SCHEEPMAKER G M , GOVERDE R M P , KROON L G . Review of energy-efficient train control and timetabling [J ] . European Journal of Operational Research , 2017 , 257 ( 2017 ): 355 - 376 .
唐伦 , 贺小雨 , 王晓 , 等 . 基于迁移演员-评论家学习的服务功能链部署算法 [J ] . 电子与信息学报 , 2020 , 42 ( 11 ): 2671 - 2679 .
TANG L , HE X Y , WANG X , et al . Deployment algorithm of service function chain based ontransfer actor-critic learning [J ] . Journal of Electronics ＆ Information Technology , 2020 , 42 ( 11 ): 2671 - 2679 .
YUAN Y L , YU Z L , GU Z H , et al . A novel multi-step Q-learning method to improve data efficiency for deep reinforcement learning [J ] . Knowledge-Based Systems , 2019 , 175 : 107 - 117 .
HEINRICH J , SILVER D . Deep reinforcement learning from self-play in imperfect-information games [J ] . arXiv preprint , 2016 ,arXiv:1603.01121.
SUTTON R , BARTO A . Reinforcement learning:an introduction [M ] . Cambridge : MIT Press , 1998 .
王飞跃 , 曹东璞 , 魏庆来 . 强化学习:迈向知行合一的智能机制与算法 [J ] . 智能科学与技术学报 , 2020 , 2 ( 2 ): 101 - 106 .
WANG F Y , CAO D P , WEI Q L . Reinforcement learning:toward action-knowledge merged intelligent mechanisms and algorithms [J ] . Chinese Journal of Intelligent Science and Technology , 2020 , 2 ( 2 ): 101 - 106 .
孙长银 , 穆朝絮 . 多智能体深度强化学习的若干关键科学问题 [J ] . 自动化学报 , 2020 , 46 ( 7 ): 1301 - 1312 .
SUN C Y , MU C X . Important scientific problems of multi-agent deep reinforcement learning [J ] . Acta Automatica Sinica , 2020 , 46 ( 7 ): 1301 - 1312 .
VOLODYMYR M , KORAY K , DAVID S , et al . Human-level control through deep reinforcement learning [J ] . Nature , 2015 , 518 ( 7540 ): 529 - 533 .
VOLODYMYR M , KORAY K , DAVID S , et al . Playing atari with deep reinforcement learning [J ] . arXiv preprint , 2013 ,arXiv:1312.5602.
张俊杰 , 张聪 , 赵涵捷 . 重复利用状态值的竞争深度Q网络算法 [J ] . 计算机工程与应用 , 2020 , 56 ( 21 ): 1 - 10 .
ZHANG J J , ZHANG C , ZHAO H J . Dueling deep Q network algorithm with state value reuse [J ] . Computer Engineering and Applications , 2020 , 56 ( 21 ): 1 - 10 .
YANG Z R , XIE Y C , WANG Z R , et al . A theoretical analysis of deep Q-learning [J ] . Learning for Dynamics and Control , 2020 : 486 - 489 .
ZHAO W Y , GUAN X Y , LIU Y , et al . Stochastic variance reduction for deep Q-learning [J ] . arXiv preprint , 2019 ,arXiv:1905.08152.
VAN HASSELT H , GUEZ A , SILVER D . Deep reinforcement learning with double Q-learning [J ] . arXiv preprint , 2015 ,arXiv:1509.06461.
OSBAND I , BLUNDELL C , PRITZEL A , et al . Deep exploration via bootstrapped DQN [J ] . Advances in Neural Information Processing Systems , 2016 : 4026 - 4034 .
HASSELT H V , WIERING M A . Reinforcement learning in continuous action spaces [C ] // 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning . Piscataway:IEEE Press , 2007 : 272 - 279 .
SU S , LI X , TANG T , et al . A subway train timetable optimization approach based on energy-efficient operation strategy [J ] . IEEE Transactions on Intelligent Transportation Systems , 2013 , 14 ( 2 ): 883 - 893 .
宿帅 . 城轨列车运行图和速度曲线一体化节能方法 [D ] . 北京:北京交通大学 , 2016 .
SU S . An energy-efficient approach by integrating the train timetable and speed trajectory in urban rail [D ] . Beijing:Beijing Jiaotong University , 2016 .
LIU W T , TANG T , SU S , et al . Energy-efficient train driving strategy with considering the steep downhill segment [J ] . Processes , 2019 , 7 ( 2 ): 77 .
GOODFELLOW I , BENGIO Y , COURVILLE A . Deep learning [M ] .[S.l. ] : MIT press , 2016 .
沈宇 , 韩金朋 , 李灵犀 , 等 . 游戏智能中的 AI-从多角色博弈到平行博弈 [J ] . 智能科学与技术学报 , 2020 , 2 ( 3 ): 205 - 213 .
SHEN Y , HAN J P , LI L X , et al . AI in game intelligence-from multi-role game to parallel game [J ] . Chinese Journal of Intelligent Science and Technology , 2020 , 2 ( 3 ): 205 - 213 .
郑南宁 . 人工智能新时代 [J ] . 智能科学与技术学报 , 2019 , 1 ( 1 ): 1 - 3 .
ZHENG N N . The new era of artificial intelligence [J ] . Chinese Journal of Intelligent Science and Technology , 2019 , 1 ( 1 ): 1 - 3 .
0
Views
269
下载量
0
CSCD
- L-PDFDownload
- XMLDownload
- Meta-XMLDownload
- BibTeXDownload
- EndnoteDownload
- RefMan Download
- NoteExpressDownload
- NoteFirstDownload
Publicity Resources
Related Articles
Related Author
Related Institution
- Postal code：100079
- Tel：（010）53879206 Email：cjist@bjxintong.com.cn
- Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19
京公网安备11010802024621 - It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
- Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
- Total Visits：35685 Visits Today：160
