Making Sense of Big Data

Spark Architecture was one of the toughest elements to grasp when initially learning about Spark. I think one of the main reasons is that there is a vast amount of information out there, but nothing which gives insight into all aspects of the Spark Ecosystem. This is most likely because, its complicated! There are so many fantastic resources out there, but not all are intuitive or easy to follow.

I am hoping this part of the series will help people who have very little knowledge of the topic understand how Spark Architecture is built, from the foundations up. …


The main driving goal of Apache Spark is to enable users to build big data applications via a unified platform in an accessible and familiar way. Spark is designed in such a way that traditional data engineers and analytical developers will be able to integrate their current skill sets, whether that be coding languages or data structures with ease.

But what does that all mean and you still haven’t answered the question!

https://www.datanami.com/2019/03/08/a-decade-later-apache-spark-still-going-strong/

Apache Spark is a computing engine which contains multiple API’s (Application Programming Interfaces), these API’s allow a user to interact using traditional methods with the back-end Spark engine…


This is a new series blogs aimed at developers in the engineering and analytical space who want to build and expand their knowledge within the inner workings of the Spark API’s (Application Programming Interfaces). A personal goal of the series is to enrich my current understanding of Apache Spark, sharing my learnings and resources in an easily digestable way, to you, the reader. So no matter what level your current understanding of distributed computing is, there will be content and material for you to use in your journey with Apache Spark. This series is aimed at Data Scientists, Data Engineers…

Luke Thorp

Learning new skills, tools and methodologies is what I do best! I am currently trying to master Apache Spark, I am not going to lie… it’s quite fun!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store