Apache Spark is a powerful open-source distributed computing system that provides an interface for programming entire clusters with implicit data parallelism and fault tolerance. It is widely used for big data processing and analytics across various domains, enabling users to handle large-scale data with ease. A critical step while working with Apache Spark is the initialization of the SparkSession and the SparkContext. They are the entry points to the functionalities of Spark, and properly understanding and managing these components is crucial for effective Spark application development.
SparkSession
- Introduced in Spark 2.0
- Provides a unified interface for working with Spark data structures, including DataFrames, Datasets, and SQL
- Simplified application development and improved productivity
- Supports built-in integration with various Spark modules
SparkContext
- The original entry point to Spark functionality
- Allows you to create RDDs, accumulators, and broadcast variables
- Provides access to Spark services and perform jobs
- Still useful for working with RDDs and low-level Spark features
Creating SparkSession
To begin working with SparkSession, one must first understand how to instantiate it within a Scala application. Below are the necessary steps to create a SparkSession and subsequently use it to perform data operations.
Prerequisites
To create SparkSession, you typically need the following prerequisites in your development environment:
- Scala programming language
- Apache Spark binaries installed or accessible in your environment
- Build tool like sbt or Maven with necessary Spark dependencies specified
For instance, your build.sbt or Maven POM file should include the Spark SQL library dependency:
// For sbt
libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.1.1"
// For Maven
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.1.1</version>
</dependency>Instantiating SparkSession
To instantiate a SparkSession, you use the SparkSession.builder method, which allows you to specify various options and configurations for your session. A simple example of creating a SparkSession in Scala looks like this:
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder
.appName("My Spark Application")
.config("spark.master", "local")
.getOrCreate()
println("SparkSession created successfully!")If you run the above Scala code and everything is configured correctly, the output would simply be:
SparkSession created successfully!The appName method names the application—it will be shown in the Spark Web UI. The config method sets a Spark property, in this case, the master URL to connect to (local mode for running on a single machine).
SparkSession Configuration Options
When building a SparkSession, you can specify additional configurations such as enabling Hive support, setting serialization properties, or tuning resource allocation for your application.
val spark = SparkSession.builder
.appName("My Spark Application")
.config("spark.master", "local")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.enableHiveSupport()
.getOrCreate()The enableHiveSupport method allows SparkSession to interact with data stored in Hive. The choice of serializer can impact the performance of your Spark job. Here, we configure the Kryo serializer, which is more efficient than the default Java serializer.
Getting Existing SparkSession or Creating New One
It is important to note that getOrCreate will either get the current active SparkSession or, if there is none, create a new one. Therefore, it safely prevents the creation of unnecessary sessions in your application.
Understanding and Creating SparkContext
Even though SparkSession has largely subsumed SparkContext, there are still scenarios where SparkContext is directly used, particularly when dealing with RDDs and lower-level APIs. SparkContext is generally accessed within a SparkSession via the sparkContext attribute.
Using SparkContext within SparkSession
Once you have a reference to an active SparkSession, you can access SparkContext directly:
val sc = spark.sparkContext
println("SparkContext accessed successfully!")The familiar sc variable now holds the active SparkContext reference, and you can proceed to perform RDD transformations and actions, or to broadcast variables and create accumulators.
Configuring SparkContext
Configuring SparkContext typically occurs during the SparkSession building process. However, if you need to create a SparkContext without SparkSession, you can do so directly:
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
val conf = new SparkConf()
.setAppName("My Spark App")
.setMaster("local")
val sc = new SparkContext(conf)
println("SparkContext created successfully!")Similar to SparkSession, you would use SparkConf to configure your SparkContext. This process is generally not necessary when using Spark 2.0 and above, as SparkSession is preferred.
SparkSession and SparkContext in spark-shell
In the spark-shell, both SparkSession and SparkContext are automatically created and are available as spark and sc, respectively. This allows you to immediately start working with Spark without the need to initialize these contexts yourself.
// In spark-shell, you can directly use `spark` and `sc`:
val df = spark.read.csv("data.csv") // Using SparkSession
val rdd = sc.textFile("data.txt") // Using SparkContextClosing SparkSession and SparkContext
Finally, it is good practice to stop your SparkSession and SparkContext when your application is finished to free up resources. Stopping SparkSession will also stop the underlying SparkContext:
spark.stop()
println("SparkSession stopped.")Which one to use?
In most cases, SparkSession is the preferred way to work with Spark. It provides a more consistent and simpler interface, and it supports built-in integration with various Spark modules. SparkContext is still useful for working with RDDs and low-level Spark features, but it can also be accessed through SparkSession if needed.
Here are some specific examples of when you might want to use SparkSession:
- If you are working with DataFrames, Datasets, or SQL
- If you want to simplify application development and improve productivity
- If you want to use built-in integration with other Spark modules
Here are some specific examples of when you might want to use SparkContext:
- If you need to use a Spark feature that is not supported by SparkSession
- If you need to fine-grained control over Spark execution
- If you are working with a legacy Spark application that uses SparkContext
In summary, creating and configuring SparkSession and SparkContext is a foundational step for any Spark application. While SparkSession is now the primary entry point, understanding SparkContext remains important for working with the lower-level APIs and legacy codebases. Properly initializing, configuring, and managing these components will ensure that your Spark applications are robust, maintainable, and tuned for performance.