Optimization of collaborative filtering algorithm based on DAG Spark scheduling

LIAO Bin; ZHANG Tao; YU Jiong; GUO Binglei; ZHANG Xuguang; LIU Yan

您当前的位置：

首页 >

文章列表页 >

Optimization of collaborative filtering algorithm based on DAG Spark scheduling

更新时间：2023-12-11

- Optimization of collaborative filtering algorithm based on DAG Spark scheduling
- Acta Scientiarum Naturalium Universitatis SunYatseni Vol. 56, Issue 3, Pages: 46-56(2017)
- 作者机构：
  
  1. 新疆财经大学统计与信息学院,新疆,乌鲁木齐,830012
  2. 
  3. 新疆大学信息科学与工程学院,新疆,乌鲁木齐,830008
- 作者简介：
- 基金信息：
- DOI：
  CLC：
- Published：2017，
  
  Published Online：25 May 2017，
扫描看全文
LIAO Bin, ZHANG Tao, YU Jiong, et al. Optimization of collaborative filtering algorithm based on DAG Spark scheduling. [J]. Acta Scientiarum Naturalium Universitatis SunYatseni 56(3):46-56(2017)
DOI：

LIAO Bin, ZHANG Tao, YU Jiong, et al. Optimization of collaborative filtering algorithm based on DAG Spark scheduling. [J]. Acta Scientiarum Naturalium Universitatis SunYatseni 56(3):46-56(2017) DOI：

摘要

大数据的规模效应给数据存储、管理以及数据分析带来了极大的挑战，高效率低成本的大数据处理技术成为学术界及工业界的研究热点。为提高协同过滤算法的执行效率，对MapReduce架构下的算法执行步骤进行了分解，并对算法执行缺陷进行了分析。结合Spark适于迭代型及交互型任务的特点，提出将算法从MapReduce平台移植Spark平台的改进思路。设计了算法在Spark中的实现流程，并通过参数调整、内存优化等方法进一步提高算法效率。实验结果表明：与MapReduce平台中的算法相比，基于Spark DAG调度的算法能够减少65%以上的HDFS重复I/O操作，执行效率与能耗效率分别提升近200%及50%。

Abstract

The scale effect of big data has brought great challenges to data storage

management and analysis. And the high efficiency and low cost big data processing technology has become a hotspot research in academia and industry. In order to improve the efficiency of collaborative filtering algorithms

the implementation of the algorithm under the MapReduce architecture is decomposed in order to analysis the defects of the algorithm. For the Spark suitable for the iterative and interactive tasks

this paper presents the methods to improve the execution efficiency from the MapReduce platform to the Spark platform. The implementation flow of the algorithm in Spark is designed

and efficiency is improved by parameter adjustment and memory optimization. Experimental results show that: based on spark DAG scheduling

the algorithm can reduce more than 65% HDFS I/O operations and enforce the efficiency and energy efficiency were increased by nearly 200% and 50%.

关键词

协同过滤MapReduceSpark算法优化能耗优化

Keywords

collaborative filteringMapReduceSparkalgorithm optimizationenergy consumption optimization

references

Views

257

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Dual Collaborative Topic Regression for Recommendation Systems

An Energy-Efficient and Heterogeneous Environment Adaptive Data Layout Strategy for MapReduce

Hybrid user and item based collaborative filtering personalized recommendation algorithm in E-commerce

Spark-based traffic flow prediction analysis using multi-order spatial weighting matrix STARIMA

Related Author

LI Gai

LI Lei

LIAO Bin

ZHANG Tao

YU Jiong

LIU Ji

ZHONG Lei

LIU Yan

Related Institution

School of Electronics and information engineering, Shunde Polytechnic, Shunde



School of Information Science and Technology, Sun Yatsen University

新疆财经大学统计与信息学院

 2 新疆大学软件学院

Postal code：510275
Tel：020-84112585，84113223 Email：xuebaozr@mail.sysu.edu.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰