Reverse-Engineering the Design Rules for Cloud-Based Big Data Platforms

Abstract

Big Data's 5 V complexities are making it increasingly difficult to develop an understanding of the end to end process. Big Data platforms play a crucial role in many critical systems, combining with Internet-of-Things, Artificial Intelligence and Business Analytics. It is both relevant and important to understand Big Data systems to identify the best tools that fit the requirements of heterogeneous platforms. The objective of this paper is to "discover" a set of design principles and rules for Cloud-based Big Data platforms for complex, heterogeneous environments. The design scope comprises Big Data's significance, challenges and architectural impacts. Using a methodology Reverse Engineered Design Science Research (REDSR), artifacts from leading vendors are used to elicit the design principles and rules with relevant details of Big Data components. We conclude that the findings are relevant and useful for DevOps architects and practitioners in operating complex, heterogeneous Cloud-based Big Data platforms.