University of Tasmania
Browse

File(s) under permanent embargo

Orchestrating big data analysis workflows in the cloud: research challenges, survey, and future directions

journal contribution
posted on 2023-05-20, 03:43 authored by Barika, M, Saurabh GargSaurabh Garg, Zomaya, AY, Wang, L, van Moorsel, A, Ranjan, R
Interest in processing big data has increased rapidly to gain insights that can transform businesses, government policies and research outcomes. This has led to advancement in communication, programming and processing technologies, including Cloud computing services and technologies such as Hadoop, Spark and Storm. This trend also affects the needs of analytical applications, which are no longer monolithic but composed of several individual analytical steps running in the form of a workflow. These Big Data Workflows are vastly different in nature from traditional workflows. Researchers are currently facing the challenge of how to orchestrate and manage the execution of such workflows. In this paper, we discuss in detail orchestration requirements of these workflows as well as the challenges in achieving these requirements. We also survey current trends and research that supports orchestration of big data workflows and identify open research challenges to guide future developments in this area.

History

Publication title

ACM Computing Surveys

Volume

52

Issue

5

Article number

95

Number

95

Pagination

1-41

ISSN

0360-0300

Department/School

School of Information and Communication Technology

Publisher

Assoc Computing Machinery

Place of publication

1515 Broadway, New York, USA, Ny, 10036

Rights statement

Copyright 2019 Association for Computing Machinery

Repository Status

  • Restricted

Socio-economic Objectives

Information systems, technologies and services not elsewhere classified; Information systems, technologies and services not elsewhere classified

Usage metrics

    University Of Tasmania

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC