What Is the Best Way to Get the Max Value in a Spark DataFrame Column?

Finding the maximum value in a column of a Spark DataFrame can be done efficiently using the `agg` (aggregate) method with the `max` function. Below, I’ll explain this using PySpark, but the concept is similar in other languages like Scala and Java. Let’s dive into the details.

Using PySpark to Get the Max Value in a Column

First, import the necessary modules and create a sample DataFrame:


from pyspark.sql import SparkSession
from pyspark.sql.functions import col, max as max_

# Initialize SparkSession
spark = SparkSession.builder.appName("MaxValueExample").getOrCreate()

# Sample DataFrame
data = [
    (1, "Alice", 50),
    (2, "Bob", 45),
    (3, "Cathy", 35),
    (4, "David", 60)
]

df = spark.createDataFrame(data, ["ID", "Name", "Score"])
df.show()

Output:


+---+-----+-----+
| ID| Name|Score|
+---+-----+-----+
|  1|Alice|   50|
|  2|  Bob|   45|
|  3|Cathy|   35|
|  4|David|   60|
+---+-----+-----+

To get the maximum value in the “Score” column, use the `agg` method with the `max` function:


max_score = df.agg(max_(col("Score")).alias("max_score")).collect()[0]["max_score"]
print("Max Score: ", max_score)

Output:


Max Score:  60

Using Scala to Get the Max Value in a Column

If you are more comfortable with Scala, the syntax is quite similar. First, create a sample DataFrame and then use the `agg` method with the `max` function:


import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

val spark = SparkSession.builder.appName("MaxValueExample").getOrCreate()

// Sample DataFrame
val data = Seq(
    (1, "Alice", 50),
    (2, "Bob", 45),
    (3, "Cathy", 35),
    (4, "David", 60)
)
val df = spark.createDataFrame(data).toDF("ID", "Name", "Score")
df.show()

Output:


+---+-----+-----+
| ID| Name|Score|
+---+-----+-----+
|  1|Alice|   50|
|  2|  Bob|   45|
|  3|Cathy|   35|
|  4|David|   60|
+---+-----+-----+

To get the maximum value in the “Score” column:


val maxScore = df.agg(max("Score").alias("max_score")).collect()(0).getAs[Int]("max_score")
println(s"Max Score: $maxScore")

Output:


Max Score: 60

Conclusion

The best way to get the maximum value in a Spark DataFrame column is to use the `agg` method with the `max` function. This method efficiently computes the maximum value, is concise, and works seamlessly across different languages supported by Spark, such as Python, Scala, and Java.

About Editorial Team

Our Editorial Team is made up of tech enthusiasts who are highly skilled in Apache Spark, PySpark, and Machine Learning. They are also proficient in Python, Pandas, R, Hive, PostgreSQL, Snowflake, and Databricks. They aren't just experts; they are passionate teachers. They are dedicated to making complex data concepts easy to understand through engaging and simple tutorials with examples.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top