| 引用本文: | 胡亚红,邱圆圆,毛家发.分布式异构集群中节点优先级调优算法.[J].国防科技大学学报,2022,44(5):102-113.[点击复制] |
| HU Yahong,QIU Yuanyuan,MAO Jiafa.Node priority optimization in distributed heterogeneous clusters[J].Journal of National University of Defense Technology,2022,44(5):102-113[点击复制] |
|
| |
|
|
| 本文已被：浏览 4493次 下载 3496次 |
| 分布式异构集群中节点优先级调优算法 |
| 胡亚红,邱圆圆,毛家发 |
|
(浙江工业大学 计算机科学与技术学院(软件学院), 浙江 杭州 310023)
|
| 摘要: |
| 节点优先级常用于评价异构集群中节点的性能,因此节点优先级评价指标权重的选择非常重要。采用层次分析法(analytic hierarchy process, AHP)建立了节点优先级评价指标体系,计算得到各指标的初始权重,并使用BP神经网络对初始权重进行优化。训练时,BP网络输入为集群运行中采集的节点实时资源数据,输出为节点的优先级。分析网络训练完成后得到的权重矩阵可以获得各优先级评价指标的优化权重。实验表明,基于AHP和BP的节点优先级评价模型可以更加准确地分析节点性能。相比于Spark默认算法和权重未优化的对照算法,使用调优后的节点优先级可以有效提高集群性能。运行不同工作量的相同负载时,集群平均性能分别提高了16.64%和9.76%；处理相同工作量的不同负载时,集群的平均性能分别提高了12.49%和6.54%。 |
| 关键词: 层次分析法 BP神经网络 节点优先级 权重 Spark |
| DOI：10.11887/j.cn.202205011 |
| 投稿日期：2020-11-22 |
| 基金项目:国家重点研发计划资助项目(2018YFB0204003) |
|
| Node priority optimization in distributed heterogeneous clusters |
| HU Yahong, QIU Yuanyuan, MAO Jiafa |
|
(College of Computer Science and Technology(College of Software), Zhejiang University of Technology, Hangzhou 310023, China)
|
| Abstract: |
| Node priority is often used to evaluate the performance of heterogeneous cluster nodes, and it is of great importance to provide suitable weight for each priority evaluation index. The AHP (analytic hierarchy process) was chosen to establish the evaluation index system of node priority, and the initial weight of each index was calculated. The BP (back propagation) neural network was then used to optimize the weights obtained by using AHP. The input of the BP neural network was the node′s performance index values collected during execution of cluster, and the output was the corresponding priority of the node. After the network training, the weight matrix was obtained and used to calculate the optimized weights. The experimental results show that the cluster node priority evaluation model based on AHP and BP can evaluate the node performance more accurately. Compared with the default resource allocation algorithm of Spark and the comparison algorithm with unoptimized weights, the cluster performance is improved effectively by using the node priority optimized. When running the same kind of load with different amount of data, the average cluster performance increases by 16.64% and 9.76%, respectively; and when running different loads with the same amount of data, the average performance of the cluster increases by 12.49% and 6.54%, respectively. |
| Keywords: analytic hierarchy process BP neural network node priority weight Spark |
|
|
|
|
|
