How to Sum a Column in a Spark DataFrame Using Scala?

Summing a column in a Spark DataFrame is a common operation you might perform during data analysis. In this example, I’ll show you how to sum a column using Scala in Apache Spark. We’ll use some simple data to demonstrate this operation.

Summing a Column in a Spark DataFrame Using Scala

First, you need to ensure that you have the Spark environment set up and running. You can do that by creating a Spark session as shown below:

import org.apache.spark.sql.SparkSession

// Create SparkSession
val spark = SparkSession.builder()
    .appName("Sum Column Example")
    .master("local[*]")
    .getOrCreate()

Next, create a DataFrame with some sample data. Let’s assume we have a DataFrame with a column named “numbers”.

import org.apache.spark.sql.functions._
import spark.implicits._

// Sample data
val data = Seq(
    (1, "a"),
    (2, "b"),
    (3, "c"),
    (4, "d"),
    (5, "e")
)

// Create DataFrame
val df = data.toDF("number", "letter")

// Show the DataFrame
df.show()

+------+------+
|number|letter|
+------+------+
|     1|     a|
|     2|     b|
|     3|     c|
|     4|     d|
|     5|     e|
+------+------+

Now, to sum the “number” column, you can use the `agg` function with the `sum` function provided by `org.apache.spark.sql.functions` package.

// Sum the "number" column
val sumColumn = df.agg(sum("number").as("sum_number"))

// Show the result
sumColumn.show()

+----------+
|sum_number|
+----------+
|        15|
+----------+

Explanation

1. **Import Dependencies**: We import necessary libraries, including `SparkSession` to create a Spark session and `functions` to use built-in functions like `sum`.

2. **Create SparkSession**: We create a `SparkSession` which is essential for any Spark application.

3. **Sample Data**: We create a sample DataFrame with a “number” column and some integer values.

4. **Sum Column**: We use the `agg` function on the DataFrame, where `sum(“number”)` calculates the sum of the “number” column. We alias this result as “sum_number” for clarity.

5. **Show Result**: Finally, we display the result using the `show` method, which shows that the sum of the “number” column is 15.

This demonstrates how easy it is to perform aggregation operations such as summing a column in a Spark DataFrame using Scala. The same concept can be extended to other aggregation functions provided by Spark.