Abstract
Efficient resource management in data centers and
clouds running large distributed data processing frameworks
like MapReduce is crucial for enhancing the performance of
hosted applications and boosting resource utilization. However,
existing resource scheduling schemes in Hadoop MapReduce
allocate resources at the granularity of fixed-size, static portions
of nodes, called slots. In this work, we show that MapReduce jobs
have widely varying demands for multiple resources, making the
static and fixed-size slot-level resource allocation a poor choice
both from the performance and resource utilization standpoints.
Furthermore, lack of co-ordination in the management of mul-
tiple resources across nodes prevents dynamic slot reconfigura-
tion, and leads to resource contention. Motivated by this, we
propose MROrchestrator, a MapReduce resource Orchestrator
framework, which can dynamically identify resource bottlenecks,
and resolve them through fine-grained, co-ordinated, and on-
demand resource allocations. We have implemented MROrches-
trator on two 24-node native and virtualized Hadoop clusters.
Experimental results with a suite of representative MapReduce
benchmarks demonstrate up to 38% reduction in job completion
times, and up to 25% increase in resource utilization. We further
show how popular resource managers like NGM and Mesos when
augmented with MROrchestrator can hike up their performance.
