Spark Jobs, Stages, Tasks
Every distributed computation is divided in small parts called jobs, stages and tasks. It’s useful to know them especially during monitoring because it helps to detect bottlenecks. Job -> Stages -> Tasks...
Tagged: Spark
Spark Interview Questions : Basic
1. What is SparkContext? “SparkContext” is the main entry point for Spark functionality. A “SparkContext” represents the connection to a Spark cluster, and can be used to create...
Spark and ElasticSearch integration
In this blog, as topic gives a glimpse what it is going to be. Here, I’m going to explain the end to end process of writing and reading data...
Missing Imputation in scala
Imputation: In statistics, imputation is the process of replacing missing data with substituted values. When substituting for a data point, it is known as “unit imputation”; when substituting...
Missing Imputation in python
Imputation: In statistics, imputation is the process of replacing missing data with substituted values. When substituting for a data point, it is known as “unit imputation”; when substituting...
Spark SQL Using Parquet
Today, I’m focusing on how to use parquet format in spark. Please get the more insight about parquet format If you are new to this format. Parquet: Apache Parquet is a...
Spark Streaming : Word Count Example
Spark Streaming makes it easy to build scalable fault-tolerant streaming applications. Spark Stream API is a near real time streaming it supports Java, Scala, Python and R....
Network Streaming in Spark
Network streaming in spark is another interesting topic, here I am going to explain how network streaming works and will provide complete spark scala code. Before jumping to the code, I...
