Interest in processing big data has increased rapidly to gain insights that can transform businesses, government policies and research outcomes. This has led to advancement in communication, programming and processing technologies, including Cloud computing services and technologies such as Hadoop, Spark and Storm. This trend also affects the needs of analytical applications, which are no longer monolithic but composed of several individual analytical steps running in the form of a workflow. These Big Data Workflows are vastly different in nature from traditional workflows. Researchers are currently facing the challenge of how to orchestrate and manage the execution of such workflows. In this paper, we discuss in detail orchestration requirements of these workflows as well as the challenges in achieving these requirements. We also survey current trends and research that supports orchestration of big data workflows and identify open research challenges to guide future developments in this area.

Item Type:	Refereed Article
Keywords:	cloud computing, big data, map reduce, workflow orchestration, requirements, approaches
Research Division:	Information and Computing Sciences
Research Group:	Distributed computing and systems software
Research Field:	Distributed systems and algorithms
Objective Division:	Information and Communication Services
Objective Group:	Information systems, technologies and services
Objective Field:	Information systems, technologies and services not elsewhere classified
UTAS Author:	Barika, M (Mr Mutaz Barika)
UTAS Author:	Garg, S (Dr Saurabh Garg)
ID Code:	132708
Year Published:	2019
Web of Science® Times Cited:	8
Deposited By:	Information and Communication Technology
Deposited On:	2019-05-17
Last Modified:	2020-07-29
Downloads:	2 View Download Statistics