A Data-Aware Scheduling Strategy for Executing Large-Scale Distributed Workflows

Task scheduling is a crucial key component for the efficient execution of data-intensive applications on distributed environments, by which many machines must be coordinated to reduce execution times and bandwidth consumption. This paper presents ADAGE, a data-aware scheduler designed to efficiently execute data-intensive workflows in large-scale computers. The proposed scheduler is based on three key features: <inline-formula> <tex-math notation="LaTeX">$i$ </tex-math></inline-formula>) <italic>critical path analysis</italic>, for discovering the critical tasks of a workflow and reducing data transferring between nodes; <inline-formula> <tex-math notation="LaTeX">$ii$ </tex-math></inline-formula>) <italic>work giving</italic>, a new dynamic planning strategy for migrating tasks from overloaded to unloaded nodes; and <inline-formula> <tex-math notation="LaTeX">$iii$ </tex-math></inline-formula>) <italic>task replication</italic>, which executes task replicas on different nodes for improving both execution time and fault tolerance. Experiments performed on a distributed computing environment composed of up to 1,024 processing nodes show that ADAGE achieves better performances than existing scheduling systems, obtaining an average reduction of up to 66&#x0025; in execution time.

Abstract licence: CC BY-NC-ND

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.