Tech Reports Home Contact
Technical Report Series
Actions
Browse Reports
Information
Information for Authors
Contact Details

Technical Report HW-MACS-TR-0096


Title Comparing Fork/Join and MapReduce
Authors Robert Stewart and Jeremy Singer
Date 2012-08-28
Abstract This paper provides an empirical comparison of fork/join and MapReduce, which are two popular parallel execution models. We use the Java fork/join framework for fork/join, and the Hadoop platform for MapReduce. We want to evaluate these two parallel platforms in terms of scalability and programmability. Our set task is the creation and execution of a simple concordance benchmark application, taken from phase I of the SICSA multicore challenge. We find that Java fork/join has low startup latency and scales well for small inputs (<5MB), but it cannot process larger inputs due to the memory limitations of a single multicore server. On the other hand, Hadoop has high startup latency (tens of seconds), but it scales to large input data sizes (>100MB) on a compute cluster. Thus we learn that each platform has its advantages, for different kinds of inputs and underlying hardware execution layers. This leads us to consider the possibility of a hybrid approach that features the best of both parallel platforms. We implement a prototype grep application using a hybrid multi-threaded Hadoop combination, and discuss its potential.
Group DSG
Notes
Download
Email Technical Report's Administrator
|MACS Home| Top of the Page

Department of Computer Science, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS, +44 (0) 131 4514152

Last Updated: 02 September 2003 Copyright Heriot-Watt University, Disclaimer