Distribution of Executors, Cores and Memory for a Spark Application
Resource Allocation is an important aspect during the execution of any spark job. If not configured correctly, a spark job can consume entire cluster resources and make other...
Author: beginnershadoop
What are workers, executors, cores in Spark Standalone cluster?
Spark uses a master/slave architecture. As you can see in the figure, it has one central coordinator (Driver) that communicates with many distributed workers (executors). The driver and...
Difference between DataFrame, Dataset, and RDD in Spark
RDD RDD is a fault-tolerant collection of elements that can be operated on in parallel. DataFrame DataFrame is a Dataset organised into named columns. It is conceptually equivalent to a...
Spark Jobs, Stages, Tasks
Every distributed computation is divided in small parts called jobs, stages and tasks. It’s useful to know them especially during monitoring because it helps to detect bottlenecks. Job -> Stages -> Tasks...
Spark Interview Questions : Basic
1. What is SparkContext? “SparkContext” is the main entry point for Spark functionality. A “SparkContext” represents the connection to a Spark cluster, and can be used to create...
Accessing Data Stored in Amazon S3 through Spark
Spark can access files in S3, even when running in local mode, given AWS credentials. By default, with s3a URLs, Spark will search for credentials in a few...
Predictive Modeling
Classification: What are the advantages of different classification algorithms? What are the advantages of using a decision tree for classification? What are the disadvantages of using a decision...
JDBC in Spark SQL
Apache Spark has very powerful built-in API for gathering data from a relational database. Effectiveness and efficiency, following the usual Spark approach, is managed in a transparent way....
Machine Learning: Logistic Regression using Apache Spark
In this blog post, I’ll help you get started using Apache Spark’s spark.ml Logistic Regression for predicting whether or not someone makes more or less than $50,000. Classification Classification...
