A survey on evolutionary ensemble learning algorithm

Yi HU; Boyang QU; Jing LIANG; Jie WANG; Yanli WANG

doi:10.11959/j.issn.2096-6652.202103

您当前的位置：

首页 >

文章列表页 >

A survey on evolutionary ensemble learning algorithm

Surveys and Prospectives | 更新时间：2024-06-05

- A survey on evolutionary ensemble learning algorithm
- Chinese Journal of Intelligent Science and Technology Vol. 3, Issue 1, Pages: 18-35(2021)
- 作者机构：
  
  1. 郑州大学电气工程学院，河南郑州 450001
  2. 中原工学院电子信息学院，河南郑州 450007
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-6652.202103
  CLC： TP181
- Published Online：2021-03，
  
  Published：15 March 2021
- 稿件说明：
移动端阅览
Yi HU, Boyang QU, Jing LIANG, et al. A survey on evolutionary ensemble learning algorithm[J]. Chinese journal of intelligent science and technology, 2021, 3(1): 18-35.
DOI：

Yi HU, Boyang QU, Jing LIANG, et al. A survey on evolutionary ensemble learning algorithm[J]. Chinese journal of intelligent science and technology, 2021, 3(1): 18-35. DOI： 10.11959/j.issn.2096-6652.202103.

摘要

进化集成学习结合了集成学习和进化算法两方面的优势，并在机器学习、数据挖掘和模式识别等领域被广泛应用。首先对进化集成学习算法的理论基础、组成结构及分类情况进行了概述。然后根据进化算法在集成学习中的优化任务，从样本选择、特征选择、集成模型参数组合优化、集成模型结构优化以及集成模型融合策略优化几个方面对当前进化集成学习领域的一些代表性研究成果进行了详细的综述，并分析和总结了各种进化集成学习算法的特点。最后对现有的进化集成研究的优缺点进行探讨，并给出了未来的研究方向。

Abstract

Evolutionary ensemble learning integrates advantages of ensemble learning and evolutionary algorithm and is widely used in machine learning

data mining

and pattern recognition.Firstly

the theoretical basis

formation

and taxonomy are introduced.Secondly

according to the optimization task of evolutionary algorithm in ensemble learning

some representative researches on evolutionary ensemble learning field were analysed from the aspects of instance selection

feature selection

parameter optimization

structure optimization

and fusion strategy optimization

and the characteristics of different evolutionary ensemble learning methods were summarized.Finally

the pros and cons of the current researches on evolutionary ensemble learning were analysed

and research directions in the future work were given.

关键词

Keywords

references

MAHMOOD A , LI T , YANG Y , et al . Semi-supervised evolutionary ensembles for web video categorization [J ] . Knowledge-Based Systems , 2015 , 76 : 53 - 66 .

NAG K , PAL N R . A multiobjective genetic programming-based ensemble for simultaneous feature selection and classification [J ] . IEEE Transactions on Cybernetics , 2015 , 46 ( 2 ): 499 - 510 .

DITTERRICH T . Machine learning research:four current direction [J ] . Artificial Intelligence Magazine , 1997 , 4 : 97 - 136 .

FERNANDEZ A , CARMONA C J , JOSE DEL JESUS M , et al . A pareto-based ensemble with feature and instance selection for learning from multi-class imbalanced datasets [J ] . International Journal of Neural Systems , 2017 , 27 ( 6 ): 1750028 .

ROSALES-PEREZ A , GARCIA S , GONZALEZ J A , et al . An evolutionary multiobjective model and instance selection for support vector machines with pareto-based ensembles [J ] . IEEE Transactions on Evolutionary Computation , 2017 , 21 ( 6 ): 863 - 877 .

姚旭 , 王晓丹 , 张玉玺 , 等 . 基于随机子空间和 AdaBoost 的自适应集成方法 [J ] . 电子学报 , 2013 , 41 ( 4 ): 810 - 814 .

YAO X , WANG X D , ZHANG Y X , et al . A self-adaption ensemble algorithm based on random subspace and AdaBoost [J ] . Acta Electronica Sinica , 2013 , 41 ( 4 ): 810 - 814 .

LIU Y , DIAO X C , CAO J J , et al . A method for entity resolution in high dimensional data using ensemble classifiers [J ] . Mathematical Problems in Engineering , 2017 ( 12 ): 1 - 11 .

李毓 , 张春霞 , 王冠伟 . 线性回归模型的 Boosting 变量选择方法 [J ] . 工程数学学报 , 2015 , 32 ( 5 ): 677 - 689 .

LI Y , ZHANG C X , WANG G W . Boosting variable selection algorithm for linear regression models [J ] . Chinese Journal of Engineering Mathematics , 2015 , 32 ( 5 ): 677 - 689 .

JIMÉNEZ F , SANCHEZ G , GARCIA J M , et al . Multi-objective evolutionary feature selection for online sales forecasting [J ] . Neurocomputing , 2017 , 234 : 75 - 92 .

HONG Y , KWONG S , CHANG Y , et al . Unsupervised feature selection using clustering ensembles and population based incremental learning algorithm [J ] . Pattern Recognition , 2008 , 41 ( 9 ): 2742 - 2756 .

CHANDRA A , YAO X . Ensemble learning using multi-objective evolutionary algorithms [J ] . Journal of Mathematical Modelling and Algorithms , 2006 , 5 ( 4 ): 417 - 445 .

CHEN H , YAO X . Multiobjective neural network ensembles based on regularized negative correlation learning [J ] . IEEE Transactions on Knowledge and Data Engineering , 2010 , 22 ( 12 ): 1738 - 1751 .

盛伟国 , 单鹏霄 . 一种基于小生境的负相关神经网络集成算法 [J ] . 浙江工业大学学报 , 2016 ( 5 ): 482 - 486 .

SHENG W G , SHAN P X . Negative correlation neural network ensemble based on a niching algorithm [J ] . Journal of Zhejiang University of Technology , 2016 ( 5 ): 482 - 486 .

SHENG W G , SHAN P , CHEN S , et al . A niching evolutionary algorithm with adaptive negative correlation learning for neural network ensemble [J ] . Neurocomputing , 2017 , 247 : 173 - 182 .

PADILHA C A , BARONE D A C , NETO A D D . A multi-level approach using genetic algorithms in an ensemble of least squares support vector machines [J ] . Knowledge-Based Systems , 2016 , 106 : 85 - 95 .

YAO X , FISCHER M , BROWN G . Neural network ensembles and their application to traffic flow prediction in telecommunications networks [C ] // Proceedings of 2001 International Joint Conference on Neural Networks . Piscataway:IEEE Press , 2001 : 693 - 698 .

JIN Y , OKABE T , SENDHOFF B . Neural network regularization and ensembling using multi-objective evolutionary algorithms [C ] // Proceedings of The 2004 Congress on Evolutionary Computation . Piscataway:IEEE Press , 2004 : 1 - 8 .

胡梦月 , 胡志坚 , 仉梦林 , 等 . 基于改进 AdaBoost.RT 和 KELM的风功率预测方法研究 [J ] . 电网技术 , 2017 , 41 ( 2 ): 536 - 542 .

HU M Y , HU Z J , ZHANG M L , et al . Research on wind power forecasting method based on improved AdaBoost.RT and KELM algorithm [J ] . Power System Technology , 2017 , 41 ( 2 ): 536 - 542 .

BUI L T , DINH T T H , et al . A novel evolutionary multi-objective ensemble learning approach for forecasting currency exchange rates [J ] . Data ＆ Knowledge Engineering , 2018 , 114 : 40 - 66 .

HANDL J , KNOWLES J . An evolutionary approach to multiobjective clustering [J ] . IEEE Transactions on Evolutionary Computation , 2007 , 11 ( 1 ): 56 - 76 .

MOHAMMADI M , NIKANJAM A , RAHMANI A . An evolutionary approach to clustering ensemble [C ] // Proceedings of the 4th International Conference on Natural Computation . Piscataway:IEEE Press , 2008 : 77 - 82 .

OBO T , KUBOTA N , LOO C K . Evolutionary ensemble learning of fuzzy randomized neural network for posture recognition [C ] // Proceedings of 2016 World Automation Congress . Piscataway:IEEE Press , 2016 : 1 - 6 .

OJHA V K , ABRAHAM A , SNASEL V . Ensemble of heterogeneous flexible neural trees using multiobjective genetic programming [J ] . Applied Soft Computing , 2017 , 52 : 909 - 924 .

MINKU L L , YAO X . Software effort estimation as a multiobjective learning problem [J ] . ACM Transactions on Software Engineering and Methodology , 2013 , 22 ( 4 ): 1 - 32 .

SMITH C , JIN Y . Evolutionary multi-objective generation of recurrent neural network ensembles for time series prediction [J ] . Neurocomputing , 2014 , 143 : 302 - 311 .

MIN S H . A genetic algorithm-based heterogeneous random subspace ensemble model for bankruptcy prediction [J ] . International Journal of Applied Engineering Research , 2016 , 11 : 2927 - 2931 .

EKBAL A , SAHA S . A multiobjective simulated annealing approach for classifier ensemble:named entity recognition in indian languages as case studies [J ] . Expert Systems with Applications , 2011 , 38 ( 12 ): 14760 - 14772 .

任克强 , 高晓林 , 谢斌 . 基于 AFSA 和 PSO 融合优化的 AdaBoost人脸检测算法 [J ] . 小型微型计算机系统 , 2016 , 37 ( 4 ): 861 - 865 .

REN K Q , GAO X L , XIE B . AdaBoost face detection algorithm based on fusion optimization of AFSA and PSO [J ] . Journal of Chinese Computer Systems , 2016 , 37 ( 4 ): 861 - 865 .

ONAN A , KORUKOGLU S , BULUT H . A multiobjective weighted voting ensemble classifier based on differential evolution algorithm for text sentiment classification [J ] . Expert Systems with Applications , 2016 , 62 : 1 - 16 .

ABRAHAM A , GROSAN C , HAN S Y , et al . Evolutionary multiobjective optimization approach for evolving ensemble of intelligent paradigms for stock market modeling [C ] // Proceedings of 2005 Mexican International Conference on Artificial Intelligence . Heidelberg:Springer-Verlag , 2005 : 673 - 681 .

CHEN Y , YANG B , ABRAHAM A . Flexible neural trees ensemble for stock index modeling [J ] . Neurocomputing , 2007 , 70 ( 4-6 ): 697 - 703 .

ALAMANIOTIS M , IKONOMOPOULOS A , TSOUKALAS L H . Evolutionary multiobjective optimization of kernel-based very-shortterm load forecasting [J ] . IEEE Transactions on Power Systems , 2012 , 27 ( 3 ): 1477 - 1484 .

YOON H S , AHN S Y , LEE S H , et al . Heterogeneous clustering ensemble method for combining different cluster results [C ] // Proceedings of the International Workshop on Data Mining for Biomedical Applications . Heidelberg:Springer-Verlag , 2006 : 82 - 92 .

LUO H , JING F , XIE X . Combining multiple clusterings using information theory based genetic algorithm [C ] // Proceedings of 2006 International Conference on Computational Intelligence and Security . Piscataway:IEEE Press , 2006 : 84 - 89 .

QIAN X , ZHANG X , JIAO L , et al . Unsupervised texture image segmentation using multiobjective evolutionary clustering ensemble algorithm [C ] // Proceedings of 2008 IEEE Congress on Evolutionary Computation . Piscataway:IEEE Press , 2008 : 3561 - 3567 .

WAHID A , GAO X , ANDREAE P . Multi-objective multi-view clustering ensemble based on evolutionary approach [C ] // Proceedings of 2015 IEEE Congress on Evolutionary Computation . Piscataway:IEEE Press , 2015 : 1696 - 1703 .

WANG Y , LIU X , XIANG L . Ga-based membrane evolutionary algorithm for ensemble clustering [J ] . Computational Intelligence and Neuroscience , 2017 .

LI X , ZHANG S , WONG K C . Single-cell RNA-seq interpretations using evolutionary multiobjective ensemble pruning [J ] . Bioinformatics , 2019 , 35 ( 16 ): 2809 - 2817 .

REN Y , ZHANG L , SUGANTHAN P N . Ensemble classification and regression-recent developments,applications and future directions [J ] . IEEE Computational Intelligence Magazine , 2016 , 11 ( 1 ): 41 - 53 .

MENDES-MOREIRA J , SOARES C , JORGE A M , et al . Ensemble approaches for regression:a survey [J ] . ACM Computing Surveys , 2012 , 45 ( 1 ): 1 - 40 .

VEGA-PONS S ， RUIZ-SHULCLOPER J . A survey of clustering ensemble algorithms [J ] . International Journal of Pattern Recognition and Artificial Intelligence , 2011 , 25 ( 3 ): 337 - 372 .

MENDES-MOREIRA J , SOARES C , JORGE A M , et al . Ensemble approaches for regression:a survey [J ] . ACM Computing Surveys , 2012 , 45 ( 1 ): 1 - 40 .

FREUND Y , SCHAPIRE R E , et al . Experiments with a new boosting algorithm [C ] // Proceedings of the 13th International Conference on Machine Learning .[S.l.:s.n. ] , 1996 : 148 - 156 .

BREIMAN L . Bagging predictors [J ] . Machine Learning , 1996 , 24 ( 2 ): 123 - 140 .

TING K M , WITTEN I H . Stacking bagged and dagged models [C ] // Proceedings of the 14th International Conference on Machine Learning.San Francisco:Morgan Kaufmann Publishers Inc . , 1997 : 367 - 375 .

HO T K . The random subspace method for constructing decision forests [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 1998 , 20 ( 8 ): 832 - 844 .

WOLPERT D H . Stacked generalization [J ] . Neural Networks , 1992 , 5 ( 2 ): 241 - 259 .

MORENO-SECO F , INESTA J M , DE LEON P J P , et al . Comparison of classifier fusion methods for classification in pattern recognition tasks [C ] // Proceedings of the Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR) . Heidelberg:Springer-Verlag , 2006 : 705 - 713 .

DIETTERICH T G , . Ensemble methods in machine learning [C ] // Proceedings of the 1st International Workshop on Multiple Classifier Systems . Heidelberg:Springer-Verlag , 2000 : 1 - 15 .

JAMES G M . Variance and bias for general loss functions [J ] . Machine Learning , 2003 , 51 ( 2 ): 115 - 135 .

KLEINBERG E . Stochastic discrimination [J ] . Annals of Mathematics and Artificial Intelligence , 1990 , 1 ( 1-4 ): 207 - 239 .

SCHAPIRE R E , FREUND Y , BARTLETT P , et al . Boosting the margin:a new explanation for the effectiveness of voting methods [J ] . The Annals of Statistics , 1998 , 26 ( 5 ): 1651 - 1686 .

FRANK E , PFAHRINGER B . Improving on bagging with input smearing [C ] // Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining . Berlin:Springer , 2006 : 97 - 106 .

BREIMAN L . Randomizing outputs to increase prediction accuracy [J ] . Machine Learning , 2000 , 40 ( 3 ): 229 - 242 .

LIN H T , LI L . Infinite ensemble learning with support vector machines [C ] // European Conference on Machine Learning . Heidelberg:Springer-Verlag , 2005 : 242 - 254 .

BI Y . The impact of diversity on the accuracy of evidential classifier ensembles [J ] . International Journal of Approximate Reasoning , 2012 , 53 ( 4 ): 584 - 607 .

张春霞 , 张讲社 . 选择性集成学习算法综述 [J ] . 计算机学报 , 2011 , 34 ( 8 ): 1399 - 1410 .

ZHANG C X , ZHANG J S . A survey of selective ensemble learning algorithms [J ] . Chinese Journal of Computers , 2011 , 34 ( 8 ): 1399 - 1410 .

SAGI O , ROKACH L . Ensemble learning:a survey [J ] . Wiley Interdisciplinary Reviews:Data Mining and Knowledge Discovery , 2018 , 8 ( 4 ): e1249 .

PACL´IK P , LANDGREBE T C , TAX D M , et al . On deriving the second-stage training set for trainable combiners [C ] // Proceedings of the 6th International Workshop on Multiple Classifier Systems . Heidelberg:Springer-Verlag , 2005 : 136 - 146 .

张春霞 . 集成学习中有关算法的研究 [D ] . 西安:西安交通大学 , 2010 .

ZHANG C X . Research on some algorithms in ensemble learning [D ] . Xi’an:Xi’an Jiaotong University , 2010 .

ROONEY N , PATTERSON D , ANAND S , et al . Dynamic integration of regression models [C ] // Proceedings of the 5th International Workshop on Multiple Classifier Systems . Heidelberg:Springer-Verlag , 2004 : 164 - 173 .

VALENTINI G , MASULLI F . Ensembles of learning machines [C ] // Proceedings of the 13th Italian Workshop on Neural Nets . Heidelberg:Springer-Verlag , 2002 : 3 - 20 .

ZHOU Z H , WU J , TANG W . Ensembling neural networks:many could be better than all [J ] . Artificial Intelligence , 2002 , 137 ( 1-2 ): 239 - 263 .

WANG Y , LI B . Multi-strategy ensemble evolutionary algorithm for dynamic multi-objective optimization [J ] . Memetic Computing , 2010 , 2 ( 1 ): 3 - 24 .

YEN G G , HE Z . Performance metric ensemble for multiobjective evolutionary algorithms [J ] . IEEE Transactions on Evolutionary Computation , 2013 , 18 ( 1 ): 131 - 144 .

QIU X , SUGANTHAN P N , AMARATUNGA G A . Ensemble incremental learning random vector functional link network for short-term electric load forecasting [J ] . Knowledge-Based Systems , 2018 , 145 : 182 - 196 .

QIU X , REN Y , SUGANTHAN P N , et al . Empirical mode decomposition based ensemble deep learning for load demand time series forecasting [J ] . Applied Soft Computing , 2017 , 54 : 246 - 255 .

YUE C , LIANG J , QU B , et al . Multimodal multiobjective optimization in feature selection [C ] // Proceedings of 2019 IEEE Congress on Evolutionary Computation . Piscataway:IEEE Press , 2019 : 302 - 309 .

YUE C , QU B , LIANG J . A multiobjective particle swarm optimizer using ring topology for solving multimodal multiobjective problems [J ] . IEEE Transactions on Evolutionary Computation , 2017 , 22 ( 5 ): 805 - 817 .

HU Y , WANG J , LIANG J , et al . A self-organizing multimodal multi-objective pigeon-inspired optimization algorithm [J ] . Science China-information Sciences , 2019 , 62 ( 7 ): 1 - 17 .

AIN Q U , SAHAF H A , XUE B , et al . A genetic programming approach to feature construction for ensemble learning in skin cancer detection [C ] // Proceedings of the 2020 Genetic and Evolutionary Computation Conference . New York:ACM Press , 2020 : 1186 - 1194 .

ZENG J , XU H Y , GONG G L , et al . An heterogeneous stacking ensemble model for laser-induced breakdown spectroscopy elemental quantitative analysis [J ] . IEEE Access , 2020 , 8 : 136141 - 136150 .

HUANG T , DUAN D T , GONG Y J , et al . Concurrent optimization of multiple base learners in neural network ensembles:an adaptive niching differential evolution approach [J ] . Neurocomputing , 2020 , 396 : 24 - 38 .

YAN S , WANG Y , KONG D , et al . Hybrid chain-hypergraph p systems for multiobjective ensemble clustering [J ] . IEEE Access , 2019 , 9 : 143511 - 143523 .

MINKU L L , YAO X . An analysis of multi-objective evolutionary algorithms for training ensemble models based on different performance measures in software effort estimation [C ] // Proceedings of the 9th International Conference on Predictive Models in Software Engineering .[S.l.:s.n. ] , 2013 : 1 - 10 .

LIU W , WU W , WANG Y , et al . Selective ensemble learning method for belief-rule-base classification system based on PAES [J ] . Big Data Mining and Analytics , 2019 , 2 ( 4 ): 306 - 318 .

FACELI K , DE CARVALHO A C , DE SOUTO M C . Multi-objective clustering ensemble [J ] . International Journal of Hybrid Intelligent Systems , 2007 , 4 ( 3 ): 145 - 156 .

LIU R , LIU Y , LI Y . An improved method for multi-objective clustering ensemble algorithm [C ] // Proceedings of the 2012 IEEE Congress on Evolutionary Computation . Piscataway:IEEE Press , 2012 : 1 - 8 .

GOUDOS S , ATHANASIADOU G E . Application of an ensemble method to uav power modeling for cellular communications [J ] . IEEE Antennas and Wireless Propagation Letters , 2019 ( 99 ): 1 .

DAI H , SHENG W . A multi-objective clustering ensemble algorithm with automatic k-determination [C ] // Proceedings of the 2019 IEEE 4th International Conference on Cloud Computing and Big Data Analysis . Piscataway:IEEE Press , 2019 : 333 - 337 .

MINKU F L , LUDERMIR T B . Clustering and co-evolution to construct neural network ensembles:an experimental study [J ] . Neural Networks , 2008 , 21 ( 9 ): 1363 - 1379 .

RAHMAN A , VERMA B . Cluster oriented ensemble classifiers using multi-objective evolutionary algorithm [C ] // Proceedings of the 2013 International Joint Conference on Neural Networks . Piscataway:IEEE Press , 2013 : 1 - 6 .

COLETTA L F , HRUSCHKA E R , ACHARYA A , et al . A differential evolution algorithm to optimise the combination of classifier and cluster ensembles [J ] . International Journal of Bio-Inspired Computation , 2015 , 7 ( 2 ): 111 - 124 .

JAN Z M , VERMA B K . Evolutionary classifier and cluster selection approach for ensemble classification [J ] . ACM Transactions on Knowledge Discovery from Data , 2019 .

JIANG F , HE J , TIAN T . A clustering-based ensemble approach with improved pigeon-inspired optimization and extreme learning machine for air quality prediction [J ] . Applied Soft Computing , 2019 , 85 : 105827 .

ONAN A , KORUKOGLU S , BULUT H . A hybrid ensemble pruning approach based on consensus clustering and multi-objective evolutionary algorithm for sentiment classification [J ] . Information Processing and Management , 2017 , 53 ( 4 ): 814 - 833 .

Views

1371

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Label iteration-based clustering ensemble algorithm

Neural architecture search for 3D model classification based on adaptive smoothness strategy

Multimodal individual emotion recognition with joint labeling based on integrated learning and clustering

Analysis and prediction of GitHub company influence based on machine learning

Sidelink resource allocation algorithm of C-V2X based on deep Q learning

Related Author

YIN Jianfei

HUANG Zhexue

YANG Jin

HE Yulin

YANG Jun

ZHOU Peng

KE Shanjun

NIE Chengyang

Related Institution

Guangdong Laboratory of Artificial Intelligence and Digital Economy

College of Computer Science and Software Engineering, Shenzhen University

School of Electronic and Information Engineering, Lanzhou Jiaotong University

Faculty of Geomatics, Lanzhou Jiaotong University

Chongqing University of Technology

Postal code：100079
Tel：（010）53879206 Email：cjist@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：35685 Visits Today：160

⁰