1. 新疆财经大学统计与信息学院,新疆,乌鲁木齐,830012
2. 2 新疆大学软件学院,新疆,乌鲁木齐,830008
3.
4. 新疆医科大学医学工程技术学院,新疆,乌鲁木齐,830011
纸质出版日期:2015,
网络出版日期:2015-11-25,
扫 描 看 全 文
廖彬, 张陶, 于炯, 等. 适应节能与异构环境的MapReduce数据布局策略[J]. 中山大学学报(自然科学版)(中英文), 2015,54(6):55-66.
LIAO Bin, ZHANG Tao, YU Jiong, et al. An Energy-Efficient and Heterogeneous Environment Adaptive Data Layout Strategy for MapReduce[J]. Acta Scientiarum Naturalium Universitatis SunYatseni, 2015,54(6):55-66.
廖彬, 张陶, 于炯, 等. 适应节能与异构环境的MapReduce数据布局策略[J]. 中山大学学报(自然科学版)(中英文), 2015,54(6):55-66. DOI:
LIAO Bin, ZHANG Tao, YU Jiong, et al. An Energy-Efficient and Heterogeneous Environment Adaptive Data Layout Strategy for MapReduce[J]. Acta Scientiarum Naturalium Universitatis SunYatseni, 2015,54(6):55-66. DOI:
大数据处理过程中产生的高能耗问题亟待解决,尤其是在数据量规模剧增的背景下。在对已有数据布局策略存在问题分析的基础上,分析了与基于存储区域划分的节能模式及与异构HDFS集群的不适应、数据块切分算法不灵活、存储节点选择的随机性等几个方面的问题,继而提出面向节能的MapReduce数据布局策略。首先,新策略适应将集群划分为不同存储区域(ActiveZone与SleepZone)的节能模式;其次,新策略对传统的数据块数计算方法进行了改进,提出作业截止时间约束下的最小任务数计算方法确定数据块数量;最后,新的存储策略增加了对异构集群环境的适应能力,并能根据不同的作业类型进行存储节点的选择。实验结果表明:新的数据布局策略能够适应异构集群环境,达到减小MapReduce作业能耗的目的。
The problem of high energy consumption producing from big data processing is an important issue that needs to be solved
especially under the background of data explosion. Based on analyzing problems of the existing data layout policy
the problems of the in adaptation of energy-saving mode based on storage area division and heterogeneous HDFS cluster
the inflexibility of data block segmentation algorithm
the randomness of storage node selection
proposing a data layout strategy orienting to energy conservation are analyzed. Firstly
the new strategy divides the cluster into two different storage areas to meet the needs of saving energy: Active-Zone and Sleep-Zone; secondly
the new strategy has made improvements on traditional data block computing method
proposes a minimum number of jobs calculation method to determine the number of data blocks; at last
the new strategy can increase the adaptability of the heterogeneous cluster environment and can choose the appropriate storage nodes according to different job types. Experimental results show that the new data layout strategy can adapt to the heterogeneous cluster environment and reach the goal of reducing energy consumption for MapReduce jobs.
绿色计算MapReduce异构环境数据布局
green computingMapReduceheterogeneous environmentdata layout
0
浏览量
250
下载量
0
CSCD
关联资源
相关文章
相关作者
相关机构